Incremental reprocessing and S3 object storage (bucket storage v4 - Service v1.26.0)

.claude/CLAUDE.md

@@ -184,6 +184,7 @@ Always use the left column. Never use the right.
 | the Service (short for the PowerSync Service) | the service |
 | Sync Rules | sync rules |
 | Sync Streams | sync streams |
+| Sync Config | sync config (except in code, e.g. `sync_config`, `sync-config.yaml`) |
 
 When referring to the PowerSync Service in shortened form, write "the Service" (capitalized), never "the service". Generic services (a Docker Compose service, a third-party service) stay lowercase.
 

architecture/powersync-service.mdx

@@ -64,9 +64,11 @@ When a change occurs in the source database that affects a certain bucket (based
 
 The PowerSync Service persists the bucket state in durable storage: there is a pluggable storage layer for bucket data, and MongoDB and Postgres are currently supported as _bucket storage_ databases. The _bucket storage_ database is separate from the connection to your _source database_ (Postgres, MongoDB, MySQL, SQL Server or Convex). Our cloud-hosting offering (PowerSync Cloud) uses MongoDB Atlas as the _bucket storage_ database.
 
+With [S3 object storage](/sync/advanced/storage-version-4#s3-object-storage) (Beta, requires storage version 4), larger blocks of prepared sync data are stored in S3 and synced to clients directly from there, while MongoDB keeps the metadata that locates them. Clients continue to connect only to the PowerSync Service.
+
 Persisting the bucket state in a database is also part of how PowerSync achieves high scalability: it means that the PowerSync Service can have a low memory footprint even as you scale to very large volumes of synced data and users/clients.
 
-The layout of that persisted data follows a storage version. That version can change when you deploy new Sync Streams or Sync Rules. That way, large internal collections can evolve without always requiring expensive upfront migrations across your entire dataset. In most cases you do not need to name the storage version in your sync config. The optional `config.storage_version` field is there for situations where you do need explicit control. Examples include preparing for a Service downgrade that only supports an older storage format, testing different storage formats in a non-production environment, or keeping bucket data on an older stable format while you change other parts of the sync config. For more details, see [Storage version](/sync/advanced/compatibility#storage-version).
+The layout of that persisted data follows a storage version. That version can change when you deploy new Sync Streams or Sync Rules. That way, large internal collections can evolve without always requiring expensive upfront migrations across your entire dataset. In most cases you do not need to name the storage version in your Sync Config. The optional `config.storage_version` field is there for situations where you do need explicit control. Examples include preparing for a Service downgrade that only supports an older storage format, testing different storage formats in a non-production environment, or keeping bucket data on an older stable format while you change other parts of the Sync Config. For more details, see [Storage version](/sync/advanced/compatibility#storage-version).
 
 
 ## Replication from the Source Database
@@ -85,7 +87,7 @@ When the PowerSync Service replicates data from the source database, it:
 
 ### Initial Replication vs. Incremental Replication
 
-Whenever a new version of Sync Streams/Rules is deployed, initial replication takes place by means of taking a snapshot of all tables/collections they reference.
+Whenever a new Sync Config is deployed, initial replication takes a snapshot of all tables or collections it references. With [incremental reprocessing](/sync/advanced/storage-version-4#incremental-reprocessing) (Beta), a Sync Streams deployment only snapshots the streams that were added or changed. Clients continue using the current copy of the data until the new one is ready.
 
 After that, data is incrementally replicated using a change data capture stream. The specific mechanism depends on the source database type: Postgres logical replication, MongoDB change streams, the MySQL binlog, SQL Server Change Data Capture, or Convex document deltas.
 

client-sdks/advanced/pre-seeded-sqlite.mdx

@@ -22,7 +22,7 @@ If you're interested in seeing an end-to-end example, we've prepared a demo repo
 ### Generate a Scoped JWT Token
 In most cases you'd want to pre-seed the SQLite database with user specific data and not all data from the source database, as you normally would when using PowerSync. For this you would need to generate JWT tokens that include the necessary properties to satisfy the conditions of the queries in your Sync Streams (or legacy Sync Rules).
 
-Let's say we have the following sync config:
+Let's say we have the following Sync Config:
 
 <Tabs>
   <Tab title="Sync Streams">

client-sdks/advanced/sequential-id-mapping.mdx

@@ -187,7 +187,7 @@ can move on to updating your Sync Streams/Sync Rules to use the UUID column inst
 
 ## Update Sync Streams
 
-As sequential IDs can only be created on the backend source database, we need to use UUIDs in the client. The sync config is updated to use the `uuid` column as the `id` column for the `lists` and `todos` tables, explicitly defining which columns to select so that `list_id` (the integer ID) is no longer exposed to the client.
+As sequential IDs can only be created on the backend source database, we need to use UUIDs in the client. The Sync Config is updated to use the `uuid` column as the `id` column for the `lists` and `todos` tables, explicitly defining which columns to select so that `list_id` (the integer ID) is no longer exposed to the client.
 
 <Tabs>
   <Tab title="Sync Streams">

configuration/powersync-service/self-hosted-instances.mdx

@@ -24,7 +24,7 @@
   # Note only 1 connection is currently supported
   # Multiple connection support is on the roadmap
   connections:
     - type: postgresql
       # The PowerSync server container can access the Postgres DB via the DB's service name.
       # In this case the hostname is pg-db
 
@@ -466,7 +466,7 @@
 </ResponseField>
 
 <ResponseField name="default_storage_version" type="number" default="2">
-  Storage version to use when deploying a sync configuration that does not specify a storage version. You typically do not need to change this. See [Storage Version](/sync/advanced/compatibility#storage-version) for the available versions and how they interact with the sync configuration's `storage_version` field. Available since Service version 1.26.0.
+  Storage version to use when deploying a Sync Config that does not specify a storage version. To opt in to [storage version 4](/sync/advanced/storage-version-4) (Beta), set `config.storage_version: 4` in your Sync Config. An explicit `config.storage_version` takes precedence over this default. Available since Service version 1.26.0.
 </ResponseField>
 
 #### MongoDB Storage
@@ -502,7 +502,9 @@
 </ResponseField>
 
 <ResponseField name="object_storage" type="object">
-  Experimental support for storing large bucket data chunks in S3-compatible object storage instead of MongoDB. Available since Service version 1.24.0. Requires storage version 3, which is not the default. Set `storage_version: 3` under the `config` block of your sync config, as described in [Storage Version](/sync/advanced/compatibility#optional-config-storage_version).
+  Stores large blocks of bucket data in Amazon S3 or an S3-compatible object store instead of MongoDB. Requires Service version 1.26.0 or later with [storage version 4](/sync/advanced/storage-version-4) (Beta), which is not currently the default. Sync Configs on version 2 keep all data in MongoDB even when this is configured. Available since Service version 1.26.0.
+
+  Follow [Self-Hosted S3 Setup](/sync/advanced/storage-version-4#self-hosted-s3-setup) to create the bucket, configure permissions and lifecycle settings, and enable object storage in the Service. The fields below describe the available configuration options.
 
   <Expandable title="properties">
     <ResponseField name="type" type="string" required>

configuration/source-db/setup.mdx

@@ -385,7 +385,7 @@ DocumentDB is not fully MongoDB-compatible. The following behaviors differ from
 - **Source database only.** DocumentDB is supported as a source database, not as PowerSync's [storage database](/architecture/powersync-service#bucket-storage) (especially relevant if you are self-hosting PowerSync). We don't currently plan to support it for storage.
 - **Post-images are not supported.** Set [Post Images](#post-images) to **Off** (`post_images: off`, the default); the `auto_configure` and `read_only` modes fail. This does not reduce correctness: DocumentDB always includes the full current document on update events, so updates and deletes still replicate correctly.
 - **Collection drop and rename are not replicated.** Already-synced rows remain in PowerSync storage under the old name. To recover, redeploy your Sync Streams to trigger a resync. Regular inserts, updates, and deletes are unaffected.
-- **Large initial snapshots may not complete on legacy storage.** DocumentDB retains only a limited amount of change-feed history. On storage versions 1 and 2, a large or busy source can age its resume position out of that window before the initial snapshot finishes, and replication restarts from scratch. Use [Storage Version 3](/sync/advanced/compatibility#storage-version) or later, which consumes the change stream while the snapshot is still running.
+- **Large initial snapshots may not complete on legacy storage.** DocumentDB keeps source changes for a limited time. On storage versions 1 and 2, PowerSync waits for the initial scan to finish before reading those changes. On a large or busy source, the earliest required changes can expire before the scan finishes, forcing PowerSync to start over. Use [storage version 4](/sync/advanced/storage-version-4), which processes new source changes while the initial snapshot runs.
 - **Rows of 15 MiB or larger are not synced.** Such rows are dropped with a logged error. This is a general [PowerSync limit](/resources/performance-and-limits), not DocumentDB-specific, but it is more reachable on DocumentDB because the source permits documents up to 16 MiB and every change event carries the full document.
 - **Large documents replicate very slowly.** DocumentDB delivers large change events much more slowly than standard MongoDB. The document still replicates correctly, but latency spikes around large documents (including updates that change only a small field), and a single large event delays every change behind it. If your workload includes large documents, validate replication latency against your cluster.
 - **Streaming updates can be delayed by up to roughly 1 second after an idle period.** DocumentDB does not currently support reliable long-polling on its change stream, so PowerSync adds a short, capped delay between polls when the stream is idle. This is a temporary workaround and is expected to be resolved in an upcoming Azure DocumentDB release.

docs.json

@@ -230,6 +230,7 @@
                   "sync/advanced/client-id",
                   "sync/advanced/case-sensitivity",
                   "sync/advanced/compatibility",
+                  "sync/advanced/storage-version-4",
                   "sync/advanced/sync-data-by-time",
                   "sync/advanced/schemas-and-connections",
                   "sync/advanced/multiple-client-versions",

integrations/neon.mdx

@@ -141,7 +141,7 @@ PowerSync uses logical replication to sync data from your Neon database.
 
 1. In the PowerSync Dashboard, select your project and instance and go to the **Sync Streams** view (shown as **Sync Rules** if using legacy Sync Rules).
 
-2. Edit the sync config in the editor and replace the contents with the below:
+2. Edit the Sync Config in the editor and replace the contents with the below:
 
 <Tabs>
   <Tab title="Sync Streams (Recommended)">
@@ -190,8 +190,8 @@ PowerSync uses logical replication to sync data from your Neon database.
   </Tab>
 </Tabs>
 
-3. Click **"Validate"** and ensure there are no errors. This validates your sync config against your Postgres database.
-4. Click **"Deploy"** to deploy your sync config.
+3. Click **"Validate"** and ensure there are no errors. This validates your Sync Config against your Postgres database.
+4. Click **"Deploy"** to deploy your Sync Config.
 
 <Tip>
 - For additional information on PowerSync's Sync Streams, refer to the [Sync Streams](/sync/streams/overview) documentation.

intro/setup-guide.mdx

@@ -224,11 +224,11 @@ PowerSync is available as a cloud-hosted service (PowerSync Cloud) or can be sel
     powersync docker configure --database postgres --storage postgres
     ```
 
-    Docker sets up Postgres for both the source database and bucket storage and creates `powersync/docker/docker-compose.yaml`. Other databases are supported as well, you will learn more about this in the next step. Before starting, replace `powersync/sync-config.yaml` with this minimal sync config:
+    Docker sets up Postgres for both the source database and bucket storage and creates `powersync/docker/docker-compose.yaml`. Other databases are supported as well, you will learn more about this in the next step. Before starting, replace `powersync/sync-config.yaml` with this minimal Sync Config:
 
     ```yaml
     config:
-      edition: 2
+      edition: 3
 
     streams:
       todos:
@@ -567,18 +567,18 @@ streams:
   </Tab>
 
   <Tab title="CLI (Cloud)">
-    Edit `powersync/sync-config.yaml` with your sync config, then validate and deploy to the linked Cloud instance:
+    Edit `powersync/sync-config.yaml` with your Sync Config, then validate and deploy to the linked Cloud instance:
 
     ```bash
     powersync validate
     powersync deploy
     ```
 
-    This deploys your full config (connection, auth, and sync config). For subsequent sync-only changes, use `powersync deploy sync-config` instead.
+    This deploys your full config (connection, auth, and Sync Config). For subsequent sync-only changes, use `powersync deploy sync-config` instead.
   </Tab>
 
   <Tab title="CLI (Self-Hosted)">
-    Edit `powersync/sync-config.yaml` with your sync config. The default file has a placeholder (`SELECT * FROM todos`). Replace it with your actual table/collection names, then apply the changes:
+    Edit `powersync/sync-config.yaml` with your Sync Config. The default file has a placeholder (`SELECT * FROM todos`). Replace it with your actual table/collection names, then apply the changes:
 
     ```bash
     powersync validate

maintenance-ops/compacting-buckets.mdx

@@ -24,6 +24,8 @@ When using PowerSync Cloud, compact logs are available in the Dashboard's **Logs
 
 For self-hosted setups (PowerSync Open Edition & PowerSync Enterprise Self-Hosted Edition), the `compact` command in the Docker image can be used to compact all buckets. This can be run manually, or on a regular schedule using Kubernetes [CronJob](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/) or similar scheduling functionality.
 
+On [storage version 4](/sync/advanced/storage-version-4), each run decides per bucket whether to merge newly added chunks, which is fast, or to run a full compaction that reads the whole bucket. Most runs are fast, so you can schedule `compact` to run every hour.
+
 [Defragmenting](/maintenance-ops/compacting-buckets#defragmenting) may still be required.
 
 ## Background
@@ -149,16 +151,10 @@ You should consider defragmenting your buckets when:
 
 ### Defragmenting Strategies
 
-There are manual and automated approaches to defragmenting:
+There are two mechanisms with different scopes:
 
-1. **Manual Defragmentation**
-   - Use the [PowerSync Dashboard](https://dashboard.powersync.com/) to manually trigger defragmentation: Select your project and instance, go to the **Settings** view, and click the **Defragment** button in the "Compact operation history" section
-   - Best for one-time cleanup or after major data changes
+1. **Update source rows, then compact.** Use this when you can identify the rows whose old `PUT` operations block compacting. You can target one table or subset of rows, but each update creates a new operation that clients must download. For recurring maintenance, schedule the updates with a tool such as `pg_cron`:
 
-2. **Scheduled Defragmentation**
-   - Set up a cron job to regularly update rows
-   - Recommended for frequently updated tables or tables with large churn
-   - Example using `pg_cron`:
    ```sql
    -- Daily defragmentation for high-churn tables
    UPDATE audit_logs SET last_updated = now() 
@@ -168,11 +164,12 @@ There are manual and automated approaches to defragmenting:
    UPDATE users SET last_updated = now() 
    WHERE last_updated < now() - interval '1 week';
    ```
-   - This will cause clients to re-sync each updated row, while preventing the number of operations from growing indefinitely. Depending on how often rows in the bucket are modified, the interval can be increased or decreased.
+
+2. **Force a full reprocess.** In the [PowerSync Dashboard](https://dashboard.powersync.com/), select your project and instance, open **Settings**, and use **Defragment** in the "Compact operation history" section. This rebuilds all sync data, even when [incremental reprocessing](/sync/advanced/storage-version-4#incremental-reprocessing) would otherwise apply, and clients download all their data again. Use it for one-time cleanup of the entire Sync Config.
 
 ### Defragmenting Trade-offs
 
-Defragmenting + compacting as described above can significantly reduce the number of operations in a bucket, at the cost of existing clients needing to re-sync that data. When and how to do this depends on the specific use-case and data update patterns.
+Both methods can significantly reduce the operation history, at the cost of existing clients downloading data again. Updating source rows and then compacting lets you control which rows are affected. The Dashboard action rebuilds and replaces the complete replication stream.
 
 Key considerations:
 1. **Frequency**: More frequent defragmentation means fewer operations per sync but more frequent re-syncs
@@ -181,10 +178,13 @@ Key considerations:
 
 ## Sync Streams Deployments
 
-Whenever modifications to [Sync Streams](/sync/streams/overview) (or legacy [Sync Rules](/sync/rules/overview)) are deployed, all buckets are re-created from scratch. This has a similar effect to fully defragmenting and compacting all buckets. This was recommended as a workaround before explicit compacting became available ([released July 26, 2024](https://releases.powersync.com/announcements/bucket-compacting)).
+Without incremental reprocessing, deploying [Sync Streams](/sync/streams/overview) recreates all buckets from scratch. This has the same effect as defragmenting and compacting every bucket, and clients download all their data again. With [incremental reprocessing](/sync/advanced/storage-version-4#incremental-reprocessing) (Beta), unchanged streams keep their buckets and operation history, so a deployment no longer clears their history.
 
-Soon, we will use [incremental reprocessing](https://github.com/orgs/powersync-ja/discussions/349) to process changed definitions only.
+Do not redeploy a Sync Config for routine history cleanup. Use compacting, targeted source-row updates, or the Dashboard's **Defragment** action instead.
 
 ## Technical Details
 
-See the [documentation](https://github.com/powersync-ja/powersync-service/blob/main/docs/storage/compacting-operations.md) in the `powersync-service` repo for more technical details on compacting.
+See the `powersync-service` repo documentation:
+
+- [Compacting operations](https://github.com/powersync-ja/powersync-service/blob/main/docs/storage/compacting-operations.md) - How compacting rewrites operation history into `MOVE` and `CLEAR` operations. This applies to all storage versions.
+- [Compaction design](https://github.com/powersync-ja/powersync-service/blob/main/docs/storage/v3-compaction-design.md) - How [storage version 4](/sync/advanced/storage-version-4) schedules chunk merges and full compactions per bucket.