docs(kits): document differences from the legacy extensions

kits/bigquery-firestore-export/CHANGELOG.md

@@ -1 +1 @@
-- Initial release
+- Initial release of kit, see README for differences between the legacy extension and this kit

kits/bigquery-firestore-export/README.md

@@ -173,6 +173,126 @@ The run document stores DTS metadata and row counts. Its `output` subcollection
 contains converted query rows. The `latest` document is updated transactionally
 so an older completion message cannot replace a newer run.
 
+## Differences from the Export BigQuery to Firestore extension
+
+This kit is version 0.2.2 of the extension repackaged as an npm package. It is a
+close port: the same two functions, the same BigQuery Data Transfer scheduled
+query, the same `transferConfigs/{configId}/runs/{runId}` and `runs/latest`
+documents, the same `WRITE_TRUNCATE` destination table naming and the same
+row-by-row copy into Firestore. Every setting keeps its extension environment
+variable name and default, so a `.env` copied from your installed instance needs
+no value changes. What changes is the instance id, the Pub/Sub topic, the identity
+the scheduled query runs as, and how repeated BigQuery columns land in Firestore.
+
+### You set `INSTANCE_ID` yourself, and the Pub/Sub topic is renamed
+
+The extension derived an instance id at install and used it to name its
+notification topic (`ext-<instance id>-processMessages`) and to tag its transfer
+config document with `extInstanceId`. Here `INSTANCE_ID` is a setting you
+provide, and it must match this instance's key in the `instances` map in
+`firebase.json`.
+
+The topic becomes `kit-<INSTANCE_ID>-processMessages`, and the kit creates it on
+first run if it does not already exist. Set `INSTANCE_ID` to your installed
+instance's id if you want the kit to adopt the scheduled query that instance
+created, because the lookup is by `extInstanceId` on the documents in
+`COLLECTION_PATH`. With a different id the kit finds nothing, creates a second
+scheduled query, and you end up with two writing into the same collection.
+
+The existing transfer config still points its notifications at the old `ext-`
+topic; the kit's update path rewrites `notification_pubsub_topic` to the new one
+on the first deploy, so the old topic can be deleted afterwards.
+
+### Repeated BigQuery columns are now written as arrays
+
+A repeated (`ARRAY`) column used to arrive in Firestore as a map keyed by
+position, `{ "0": ..., "1": ... }`, because the conversion treated every
+non-scalar value as an object. The kit writes a real Firestore array instead.
+Anything reading those fields by numeric string key needs updating, and rows
+written before and after the change are not the same shape. Scalars, timestamps,
+dates, times, datetimes, bytes and geography values convert exactly as before.
+
+### The scheduled query runs as a different service account
+
+The extension created the transfer config with
+`serviceAccountName: ext-<instance id>@<project>.iam.gserviceaccount.com`, so the
+query ran as the extension's own service account. The kit creates it without a
+service account name, so BigQuery Data Transfer runs it as the identity that
+created it, which is your function's runtime service account (the default compute
+service account unless you have set one).
+
+That account needs to be able to read whatever `QUERY_STRING` touches and write
+to `DATASET_ID`. `roles/bigquery.admin` on the function covers this for datasets
+in the same project; a cross-project query needs the grant made explicitly. This
+was not exercised against a live deploy.
+
+Note also that a service account cannot be changed on an existing transfer config,
+so a scheduled query originally created by an installed extension instance keeps
+running as the extension's service account even after the kit adopts it. Delete
+and recreate the scheduled query if you want it moved.
+
+### The setup step runs on every deploy
+
+The install, update and configure hooks are replaced by an `upsertTransferConfig`
+task that the CLI runs after your first deploy and after every redeploy. It does
+the same work: create the scheduled query if this instance has none, otherwise
+reconcile the existing one against your current `QUERY_STRING`, `DATASET_ID`,
+`TABLE_NAME`, `PARTITIONING_FIELD` and `SCHEDULE`.
+
+Two consequences of it now being an ordinary task rather than a lifecycle event.
+There is no install UI to report progress into, so failures show up in the task's
+function logs, and the task retries up to five times with a 30 second minimum
+backoff. And `DISPLAY_NAME` is no longer immutable, but changing it does not
+rename an existing scheduled query, because display name is not part of the
+update; it only applies to a config the kit creates.
+
+Removing `PARTITIONING_FIELD` once it has been set still fails, with the same
+explanation, because the BigQuery Data Transfer API cannot clear it.
+
+### You can link an existing scheduled query
+
+`TRANSFER_CONFIG_NAME` is a new setting. Point it at the full resource name of a
+scheduled query you already have
+(`projects/<project>/locations/<location>/transferConfigs/<id>`) and the kit
+records that config in Firestore and consumes its notifications instead of
+creating one of its own. The extension carried the code for this but no setting to
+reach it. Leave it empty for the create-or-reconcile behaviour described above.
+
+### Region, and no location setting
+
+`LOCATION` is gone. Both functions deploy to your codebase's default region
+(`us-central1` unless you have changed it) rather than the immutable location you
+picked at install. `BIGQUERY_DATASET_LOCATION` is unchanged and still tells the
+result query where your dataset lives.
+
+### Failed notifications are retried
+
+`processMessages` is a 2nd gen Pub/Sub function with retries enabled, where the
+extension's 1st gen trigger did not retry. A run whose results fail to copy, for
+example because BigQuery or Firestore is briefly unavailable, is now retried
+rather than dropped. A notification that keeps failing, such as one for a transfer
+config not tagged with this `INSTANCE_ID`, is also retried until Pub/Sub gives up.
+
+Both functions' service accounts need `roles/eventarc.eventReceiver` and
+`roles/run.invoker` on top of the three roles the extension asked for, and the
+Pub/Sub API is now requested explicitly. The Firebase CLI handles all of this.
+
+### Unchanged
+
+- `COLLECTION_PATH` still defaults to `transferConfigs`, and the document layout
+  under it is identical: the transfer config document keyed by config id, a `runs`
+  subcollection keyed by run id holding `runMetadata`, `totalRowCount` and
+  `failedRowCount`, a `latest` document, and an `output` collection of rows per
+  run.
+- The destination table is still `TABLE_NAME_{run_time|"%H%M%S"}` with
+  `WRITE_TRUNCATE`, and results are still read with `SELECT *` in
+  `BIGQUERY_DATASET_LOCATION`.
+- Runs that do not succeed still write a run document with zeroed counts and still
+  update `latest`, and `latest` is still only moved forward by a newer run.
+- Rows are still written one document at a time in chunks of 10,000, with
+  per-row failures logged and counted rather than aborting the run.
+- `LOG_LEVEL` still accepts `debug`, `info`, `warn`, `error` and `silent`.
+
 ## API surface
 
 - **Main entry** (`@firebase/bigquery-firestore-export`): exports

kits/delete-user-data/CHANGELOG.md

@@ -1 +1 @@
-- Initial release
+- Initial release of kit, see README for differences between the legacy extension and this kit

kits/delete-user-data/README.md

@@ -135,6 +135,75 @@ When `EVENTARC_CHANNEL` is configured, the functions publish deletion events
 for each backend under `firebase.extensions.delete-user-data.v1.*`
 (`firestore`, `database`, and `storage`).
 
+## Differences from the Delete User Data extension
+
+This kit is the extension repackaged as an npm package, but a few things behave
+differently. If you are moving from an installed extension instance, read this
+section before you deploy.
+
+### Auto-discovery uses `true` / `false`
+
+`ENABLE_AUTO_DISCOVERY` is a boolean param, and only the literal string `true`
+enables it. The extension used `yes` / `no`, so copying an old config across
+leaves auto-discovery silently switched off. Change `yes` to `true` in your
+`.env`.
+
+### You set `INSTANCE_ID` yourself
+
+The extension derived an instance id at install time and used it to name the
+Pub/Sub topics. Here it is a setting you provide, and it must match this
+instance's key in the `instances` map in `firebase.json`. If the two disagree,
+auto-discovery publishes to a topic nothing is listening on.
+
+### Pub/Sub topics are named differently
+
+Discovery and deletion topics are now `kit-<INSTANCE_ID>-discovery` and
+`kit-<INSTANCE_ID>-deletion`, where the extension used an `ext-` prefix. The
+Firebase CLI creates them for you on deploy, so there is no manual setup step,
+but the old topics from an extension install are not reused and can be deleted
+once you have migrated.
+
+You can also override both names with `DISCOVERY_TOPIC_NAME` and
+`DELETION_TOPIC_NAME`, which the extension did not allow. Change them together,
+since one function publishes to a topic the other is triggered by.
+
+### Realtime Database deletion no longer needs a database instance
+
+The extension only cleared RTDB paths when both `SELECTED_DATABASE_INSTANCE`
+and `SELECTED_DATABASE_LOCATION` were set. This kit clears them whenever
+`RTDB_PATHS` is set, falling back to your project's default database when no
+instance is named. Set `SELECTED_DATABASE_INSTANCE` explicitly if you are
+targeting a secondary database, and leave `RTDB_PATHS` empty if you do not want
+RTDB touched at all.
+
+### Functions deploy to your default region
+
+The extension deployed to the location you picked at install time. This kit
+sets no region, so its functions deploy to your codebase's default
+(`us-central1` unless you have changed it).
+
+### Pub/Sub handlers are 2nd gen
+
+`handleSearch` and `handleDeletion` are now 2nd gen functions. `clearData`
+stays 1st gen, because the Firebase Auth `user.delete` trigger has no 2nd gen
+equivalent. This mainly matters if you have infrastructure or alerting keyed to
+function generation.
+
+### Empty search fields no longer error
+
+Setting `AUTO_DISCOVERY_SEARCH_FIELDS` to an empty value used to raise an
+invalid field path error during discovery. It is now treated as "match on the
+document path only". The default is unchanged (`id,uid,userId`).
+
+### Unchanged
+
+Events are the same. When `EVENTARC_CHANNEL` is configured, the functions still
+publish `firebase.extensions.delete-user-data.v1.firestore`, `.database` and
+`.storage` with the same payloads. Path syntax (`{UID}` substitution, comma
+separated lists, `{DEFAULT}` for the default Storage bucket), the shallow and
+recursive Firestore delete modes, the search depth and field matching rules,
+and the custom `SEARCH_FUNCTION` contract all behave as they did.
+
 ## API surface
 
 - **Main entry** (`@firebase/delete-user-data`): exports `clearData`,

kits/firestore-bigquery-export/CHANGELOG.md

@@ -1 +1 @@
-- Initial release
+- Initial release of kit, see README for differences between the legacy extension and this kit

kits/firestore-bigquery-export/README.md

@@ -215,6 +215,75 @@ missing when a write arrives, the inline write fails, the handler calls
 failure is surfaced to the function runtime retry policy (`retry: true` on
 `fsexportbigquery`).
 
+## Differences from the Stream Firestore to BigQuery extension
+
+This kit is the extension repackaged as an npm package, but a few things behave
+differently. If you are moving from an installed extension instance, read this
+section before you deploy.
+
+### Boolean settings use `true` / `false`
+
+`WILDCARD_IDS`, `USE_NEW_SNAPSHOT_QUERY_SYNTAX` and `EXCLUDE_OLD_DATA` are
+boolean params, and only the literal string `true` enables them. The extension
+used `yes` / `no` for the last two, so copying an old config across leaves them
+silently disabled. Change any `yes` to `true` in your `.env`.
+
+### Failed writes retry differently
+
+The extension pushed a failed BigQuery write onto a Cloud Tasks queue
+(`syncBigQuery`) and retried it from there. This kit has no task queue on the
+write path. A failed write is retried once in place, and anything still failing
+is handed to the Cloud Functions runtime retry policy, which redelivers the
+Firestore event.
+
+The practical effects: retries no longer show up as a separate function or
+queue in the console, and the two knobs that tuned that queue,
+`MAX_DISPATCHES_PER_SECOND` and `MAX_ENQUEUE_ATTEMPTS`, no longer exist.
+
+### Events
+
+`onSuccess` is no longer published. The extension emitted it from the task
+queue handler, which is gone, so the kit publishes `onStart` and `onError`
+only.
+
+Events are published under `firebase.extensions.firestore-bigquery-export.v1.*`
+only. The extension also published a duplicate copy of every event under
+`firebase.extensions.firestore-counter.v1.*`, a historical naming mistake kept
+for backwards compatibility. If you have Eventarc triggers listening on those
+`firestore-counter` types, point them at the `firestore-bigquery-export` types.
+
+### Wildcard columns include the document ID
+
+With `WILDCARD_IDS=true`, the wildcard column now contains a `documentId` key
+alongside the path parameters from your collection path. The extension wrote
+the path parameters only.
+
+### Functions deploy to your Firestore region
+
+The extension let you pick a function location separately from the Firestore
+database location. Here, `DATABASE_REGION` sets both: the trigger, the
+lifecycle tasks, and the database being watched.
+
+### Defaults
+
+Two settings now have defaults rather than being passed through empty:
+`DATASET_LOCATION` defaults to `us`, and `BIGQUERY_PROJECT_ID` defaults to the
+project the functions are deployed to.
+
+### Tooling that is not included
+
+The extension shipped companion scripts that this package does not:
+
+- `fs-bq-import-collection`, for backfilling documents that already existed
+  before the export started.
+- `gen-schema-view`, for generating strongly typed BigQuery views over the
+  changelog.
+- The cross-project access grant scripts.
+
+`IMPORT_COLLECTION_PATH` is not a setting here. If you rely on any of these,
+keep using the versions from the extension repository. They operate on the same
+BigQuery changelog table, so they still work against data this kit writes.
+
 ## API surface
 
 - **Main entry** (`@firebase/firestore-bigquery-export`): exports

kits/firestore-bundle-builder/CHANGELOG.md

@@ -1 +1 @@
-- Initial release
+- Initial release of kit, see README for differences between the legacy extension and this kit

kits/firestore-bundle-builder/README.md

@@ -119,6 +119,98 @@ Instance ids must be unique across all kit stanzas in the project, and every
 instance's function names are namespaced by its `kit-<instance id>-` prefix, so
 the instances cannot collide.
 
+## Differences from the Firestore Bundle Builder extension
+
+This kit is the extension repackaged as an npm package. Config is a
+lift-and-shift (`BUNDLESPEC_COLLECTION`, `BUNDLE_STORAGE_BUCKET` and
+`STORAGE_PREFIX` keep their names, defaults and meanings), but several
+behaviours changed. If you are moving from an installed extension instance,
+read this section before you deploy.
+
+### Bundle specs are read per request
+
+The extension opened a snapshot listener on the whole spec collection at
+startup and served every request from an in-memory copy. This kit reads the
+spec document directly on each request.
+
+Three consequences:
+
+- Spec edits take effect immediately, with no dependence on listener delivery,
+  and there is no cold-start window where a request waits for the first
+  snapshot.
+- A deleted spec now returns 404. The extension kept serving it, because
+  entries were only ever added to the in-memory map, never removed.
+- Each request costs one document read. If you serve high volumes of
+  uncacheable bundles, budget for that.
+
+### The Cloud Storage cache behaves differently on a miss
+
+When a spec sets `fileCache`, the extension asked Cloud Storage for a read
+stream without first checking the object existed. A missing object failed
+asynchronously, after the response was already being written. This kit confirms
+the object exists before streaming, and falls through to rebuilding the bundle
+when it does not.
+
+Failures writing the built bundle back to Cloud Storage are now logged rather
+than left unhandled. The response is still served from the freshly built
+bundle.
+
+### `fileCache` is not a time-to-live
+
+Worth stating plainly, since the name suggests otherwise: a cached bundle is
+served regardless of age. `fileCache` controls *whether* a bundle is cached,
+not for how long. This matches the extension, which accepted a `ttlSec` value
+and never enforced it. Use `clientCache` and `serverCache` for cache-control
+headers if you need expiry.
+
+### Path parameters are validated
+
+Parameter values substituted into a bundle spec's document or collection path
+are now rejected if they contain a `/`, or if they resolve to an empty value.
+Both cases return an invalid-argument error and are logged. The extension
+substituted them as-is, which allowed a caller to reach a path the spec author
+did not intend.
+
+If a spec legitimately relies on a parameter expanding to a multi-segment path,
+it will now fail. Split it into separate parameters, one per path segment.
+
+### Requests without a bundle ID return 404
+
+A request to the function root, or with a trailing slash, returns 404 with the
+usual "could not find bundle" message. The extension returned a 500 in that
+case. Note the ID is the last path segment, not a query parameter, so `?id=x`
+has never selected a bundle.
+
+### Comma-separated values for `in` queries
+
+A query condition using `in` or `not-in` accepts a comma-separated string and
+splits it. Non-string values are now passed through unchanged rather than
+being forced through string splitting, which used to throw.
+
+### Region
+
+`serve` deploys to `us-central1`, the same region the extension pinned. This is
+fixed by the package rather than chosen at install time.
+
+### Disabling the Storage cache
+
+Setting `BUNDLE_STORAGE_BUCKET` to an empty value disables the Storage cache
+outright, and specs with `fileCache` are built fresh on every request. The
+default is your project's default Storage bucket, as before.
+
+### The admin dashboard is not included
+
+The extension shipped a separate Remix admin dashboard for authoring bundle
+specs. It is not part of this package. Bundle spec documents are ordinary
+Firestore documents, so you can keep using the dashboard from the extension
+repository against the same collection, or write the documents yourself.
+
+### Runtime
+
+The functions run on Node 22 with 2nd gen Cloud Functions, where the extension
+was on Node 14 with 1st gen. Bundle format and the client-side APIs for loading
+bundles are unaffected.
+
 ## API surface
 
 - **Main entry** (`@firebase/firestore-bundle-builder`): exports `serve`. The

kits/firestore-counter/CHANGELOG.md

@@ -1 +1 @@
-- Initial release
+- Initial release of kit, see README for differences between the legacy extension and this kit