fix(metadata-protocol): bump write epoch on remote metadata mutation receipt

.changeset/metadata-protocol-remote-mutation-epoch-bump.md

@@ -0,0 +1,35 @@
+---
+'@objectstack/metadata-protocol': patch
+---
+
+Fix: a peer replica's `meta-overlay-cache` no longer re-serves a datasource (or
+any overlay row) a cluster peer just deleted
+
+`applyRemoteMetadataMutation` — the receipt side of the `metadata.mutated`
+cluster channel (#13331) — converged a peer replica's in-memory registry
+correctly, but performed no local engine write of its own, so the peer's
+`meta-overlay-cache` write-epoch never moved and its pre-mutation row set
+stayed "fresh" for the rest of its TTL (default 30s,
+`OS_METADATA_OVERLAY_CACHE_TTL_MS`). A single `GET /api/v1/meta/:type` read of
+that replica's own door, landing inside that residue window, then ran
+`hydrateOverlayIntoRegistry` over the stale rows and wrote the just-deleted
+entry straight back into the registry the bridge had just healed — and the
+registry itself carries no TTL, so that one read converted a bounded ~30s
+residue into an unbounded one for the rest of the process's life.
+
+`applyRemoteMetadataMutation` now retires this replica's overlay-cache entries
+at the moment of convergence — after the registry-convergence branch and
+before `notifyMutationListenersLocal` (the #5109 invalidate-before-notify
+rule) — via a new structural helper, `bumpWriteEpoch`, declared beside the
+existing `readWriteEpoch` in `meta-overlay-cache.ts`. This package must not
+import `@objectstack/objectql`, so the bump is spelled the same
+feature-detected way `readWriteEpoch` already is, never as a direct import of
+the epoch type. The metadata cluster channel now gets the same write-epoch
+bump the authorization cluster channel already had
+(`authz-invalidation-bridge.ts`'s `epoch.bump('remote')`, on the identical
+substrate) — closing an asymmetry between the two, not adding a new mechanism.
+
+No public API changes: `bumpWriteEpoch` is package-internal (not re-exported
+from `src/index.ts`, matching `meta-overlay-cache.ts`'s existing
+`metaOverlayCacheEntryCount`), and its one caller is the existing
+`applyRemoteMetadataMutation` receipt path.

content/docs/concepts/metadata-lifecycle.mdx

@@ -190,11 +190,17 @@ The hash is `sha256:` + 64-hex of a canonical (sorted-keys, no-undefined) JSON s
 > before its own middleware chain runs, so read-your-writes is exact on the node
 > that made the write. A **peer's** write does not: `metadata.changed` invalidates
 > the MetadataManager caches this note is about, but it does not retire the
-> overlay-read cache. What retires that on a peer is the `authz.invalidated`
-> channel — a hint from another node bumps the local write epoch — or, failing
-> that, `OS_METADATA_OVERLAY_CACHE_TTL_MS` (default 30s, `0` disables the cache
-> outright). So on a deployment with no distributed cluster driver attached, a
-> peer's overlay re-read can lag a remote publish by up to that TTL. See
+> overlay-read cache. What retires that on a peer is either of two receipt
+> paths that bump the same local write epoch: the `metadata.mutated`
+> channel's own — `protocol.ts`'s `applyRemoteMetadataMutation` calls
+> `bumpWriteEpoch(engine, 'remote')` right after its registry converges and
+> before local listeners run (#13609, 2026-09-03) — or the `authz.invalidated`
+> channel's bridge, which bumps the same epoch for the authorization cache
+> and retires this one too, as a side effect of sharing it. Failing both, the
+> floor is `OS_METADATA_OVERLAY_CACHE_TTL_MS` (default 30s, `0` disables the cache
+> outright). So only on a deployment with no cluster bridge attached at all —
+> the in-process `memory` driver, with no distributed driver behind it — does
+> a peer's overlay re-read still lag a remote publish by up to that TTL. See
 > [Environment variables](/docs/deployment/environment-variables).
 
 ---

content/docs/deployment/environment-variables.mdx

@@ -361,8 +361,8 @@ the hosted ObjectOS Cloud control plane.
 | `OS_SANDBOX_HOOK_TIMEOUT_MS` | number | `250` | Default **CPU-time** budget for a sandboxed **hook** body (QuickJS, ADR-0102): how much *VM-active* time a body may burn — idle host-await time and a nested hook's own run are NOT charged. A loaded/slow host rarely needs to raise this now (it is not wall-clock), but the knob remains. Only a positive integer is honored; unset / non-numeric / non-positive keeps the 250ms default. A hook body's own declared `timeoutMs` still wins over this. |
 | `OS_SANDBOX_ACTION_TIMEOUT_MS` | number | `5000` | Default **CPU-time** budget for a sandboxed **action** body (QuickJS). Same resolution rules as the hook variant above (positive integer only; an action body's own `timeoutMs` still wins). |
 | `OS_SANDBOX_WALL_CEILING_MS` | number | `30000` | Wall-clock ceiling (ADR-0102) — the backstop that cuts a hook/action body stuck on a host call that never settles (which burns no CPU, so the CPU budget alone would never fire). The effective ceiling is `max(this, cpuBudget)`, so it can never cut a body still inside its CPU budget. Positive integer only; unset keeps 30s. |
-| `OS_LOCALIZATION_CACHE_TTL_MS` | number | `30000` | Staleness bound, in milliseconds, for the cross-request cache of a workspace's reference localization (`timezone` / `locale` / `currency`, read from `sys_setting`) — leg C of #11633. `0` means **off**, a real path that restores the uncached query pattern exactly. Unlike `OS_AUTHZ_GRANTS_CACHE_TTL_MS` (which is off by default) this one ships **on**, because its invalidation is synchronous and in-process rather than TTL-bound: a `localization` settings change and any engine write both retire a cached answer immediately, so the TTL only bounds what neither seam can see — a write made on another replica with no `authz.invalidated` bridge attached. ⚠️ A malformed value reads as `0` (off), the opposite arm from the grants variable and deliberately so: there `0` is also the default, whereas here folding `3OOO` (letter O) into the default would hand you a **longer** window than the one you were setting. Deployment config only — never a settings row, because `sys_setting` is the table this cache caches. |
-| `OS_METADATA_OVERLAY_CACHE_TTL_MS` | number | `30000` | Staleness bound, in milliseconds, for the cross-request cache of the `sys_metadata` overlay read inside `getMetaItems` — leg D of #11633. `0` means **off**, a real path that restores the uncached query pattern exactly. Ships **on**, for the same reason as `OS_LOCALIZATION_CACHE_TTL_MS`: invalidation is synchronous and in-process, because every `sys_metadata` write goes through the engine and so advances the write epoch that retires the entry before the next read. What is cached is the overlay ROW SET only — never the merged answer — so the SchemaRegistry, the MetadataService and the artifact table are re-consulted on every call, cached or not, and the read-side registry hydration keeps running on a cache hit. The TTL therefore bounds one thing: a write made on **another replica** with no `authz.invalidated` bridge attached. ⚠️ A malformed value reads as `0` (off) — same arm and same reason as `OS_LOCALIZATION_CACHE_TTL_MS`. Deployment config only, never a settings row. |
+| `OS_LOCALIZATION_CACHE_TTL_MS` | number | `30000` | Staleness bound, in milliseconds, for the cross-request cache of a workspace's reference localization (`timezone` / `locale` / `currency`, read from `sys_setting`) — leg C of #11633. `0` means **off**, a real path that restores the uncached query pattern exactly. Unlike `OS_AUTHZ_GRANTS_CACHE_TTL_MS` (which is off by default) this one ships **on**, because its invalidation is synchronous and in-process rather than TTL-bound: a `localization` settings change and any engine write both retire a cached answer immediately, so the TTL only bounds what neither seam can see — a write made on another replica with no cluster bridge attached at all: neither the `metadata.mutated` channel's receipt path (`applyRemoteMetadataMutation` calls `bumpWriteEpoch`, #13609) nor the `authz.invalidated` bridge, both of which bump this same epoch when attached. ⚠️ A malformed value reads as `0` (off), the opposite arm from the grants variable and deliberately so: there `0` is also the default, whereas here folding `3OOO` (letter O) into the default would hand you a **longer** window than the one you were setting. Deployment config only — never a settings row, because `sys_setting` is the table this cache caches. |
+| `OS_METADATA_OVERLAY_CACHE_TTL_MS` | number | `30000` | Staleness bound, in milliseconds, for the cross-request cache of the `sys_metadata` overlay read inside `getMetaItems` — leg D of #11633. `0` means **off**, a real path that restores the uncached query pattern exactly. Ships **on**, for the same reason as `OS_LOCALIZATION_CACHE_TTL_MS`: invalidation is synchronous and in-process, because every `sys_metadata` write goes through the engine and so advances the write epoch that retires the entry before the next read. What is cached is the overlay ROW SET only — never the merged answer — so the SchemaRegistry, the MetadataService and the artifact table are re-consulted on every call, cached or not, and the read-side registry hydration keeps running on a cache hit. The TTL therefore bounds one thing: a write made on **another replica** with no cluster bridge attached at all — neither the `metadata.mutated` channel's own receipt path (`applyRemoteMetadataMutation` calls `bumpWriteEpoch` right after registry convergence, #13609) nor the `authz.invalidated` bridge, either of which retires this entry the moment it is attached and fires. ⚠️ A malformed value reads as `0` (off) — same arm and same reason as `OS_LOCALIZATION_CACHE_TTL_MS`. Deployment config only, never a settings row. |
 | `OS_INLINE_SEED_BUDGET_MS` | number | `8000` | Time budget for synchronous seed execution at boot before deferring to a worker. |
 | `OS_TENANT_AUDIT` | flag | `1` | Set to `0` to silence the tenant-isolation audit warnings emitted by the SQL driver. |
 

content/docs/permissions/system-context.mdx

@@ -112,7 +112,7 @@ that silently does not happen.
 | 18 | **`readonly` strip bypassed — UPDATE, single row** | objectql | Get: a `readonly` field CAN be written. Lose: the protection that stops a caller seeding e.g. `approval_status` | `objectql/src/engine.ts:11290` |
 | 19 | **`readonly` strip bypassed — UPDATE, bulk/predicate** | objectql | Same, on the multi-row path | `objectql/src/engine.ts:11473` |
 | 20 | **`readonly` strip bypassed — INSERT (engine pass)** | objectql | Same, on create | `objectql/src/engine.ts:10025` |
-| 21 | **`readonly` strip bypassed — INSERT (protocol ingress)** | metadata-protocol | `isSystem` is the **only** exemption here. `preserveAudit` is deliberately not read on this path (#6640) — a non-system historical import is still stripped on create | `metadata-protocol/src/protocol.ts:1746` |
+| 21 | **`readonly` strip bypassed — INSERT (protocol ingress)** | metadata-protocol | `isSystem` is the **only** exemption here. `preserveAudit` is deliberately not read on this path (#6640) — a non-system historical import is still stripped on create | `metadata-protocol/src/protocol.ts:1747` |
 | 22 | Strict-drop refusal never fires | objectql | Lose: a caller that opted into loud refusal gets **silence** — strict refuses exactly what the strip would have taken, and the strip took nothing | `objectql/src/engine.ts:10073`, `readonly-strict-errors.ts:66` |
 | 23 | **Referential-integrity check skipped** | objectql | Get: writes proceed against unreachable/unresolvable targets. Lose: an `isSystem` caller can write a **dangling reference** | `objectql/src/engine.ts:5892` |
 | 24 | Tenant-audit warning silenced; `bypassTenantAudit` threaded to the driver | objectql | Get: unscoped system writes stop warning. Lose: the signal that would flag a genuine user-path scoping bug | `objectql/src/engine.ts:3736`, `:3746`, `:3773` |

packages/metadata-protocol/src/meta-overlay-cache.test.ts

@@ -84,6 +84,7 @@
 import { describe, expect, it } from 'vitest';
 import { ObjectStackProtocolImplementation } from './protocol.js';
 import {
+    bumpWriteEpoch,
     META_OVERLAY_CACHE_DEFAULT_TTL_MS,
     metaOverlayCacheEntryCount,
     metaOverlayCacheTtlMs,
@@ -385,6 +386,53 @@ describe('[#11967] §3 a success is cached ONLY when the engine exposes the writ
     });
 });
 
+// ═══════════════════════════════════════════════════════════════════════════
+// [#13609] bumpWriteEpoch — the structural sibling that retires this cache
+// from OUTSIDE a local engine write, mirroring `authz-invalidation-bridge.ts`'s
+// `epoch.bump('remote')` on the identical substrate. `protocol.ts`'s
+// `applyRemoteMetadataMutation` is the one call site (see
+// `protocol.datasource-delete-prolongation.test.ts` for the end-to-end
+// measurement); this block pins the helper itself, the same way §3 above pins
+// `readWriteEpoch` apart from any one caller.
+// ═══════════════════════════════════════════════════════════════════════════
+
+describe('[#13609] bumpWriteEpoch — the OUTSIDE-a-write invalidation seam', () => {
+    it('bumps the seam and returns its new value when the engine exposes one', () => {
+        const engine = { writeEpoch: makeEpochSeam() };
+        expect(readWriteEpoch(engine)).toBe(0);
+        expect(bumpWriteEpoch(engine, 'remote')).toBe(1);
+        expect(readWriteEpoch(engine)).toBe(1);
+    });
+
+    it('declines the same way readWriteEpoch does — no seam, or only a partial one', () => {
+        expect(bumpWriteEpoch({ writeEpoch: { current: 3 } }, 'remote')).toBeUndefined();
+        expect(bumpWriteEpoch({ writeEpoch: { current: 3, bump: () => 4 } }, 'remote')).toBeUndefined();
+        expect(bumpWriteEpoch({}, 'remote')).toBeUndefined();
+        expect(bumpWriteEpoch(null, 'remote')).toBeUndefined();
+    });
+
+    it('retires a live cache entry the same way a local engine write does', async () => {
+        const h = makeHarness(clone(OVERLAY_ROWS));
+
+        await h.protocol.getMetaItems({ type: 'object' });
+        const perCall = h.finds.length;
+        expect(perCall).toBeGreaterThan(0);
+
+        // A repeat still hits — the control half of this assertion, paired per
+        // this file's own header rule.
+        await h.protocol.getMetaItems({ type: 'object' });
+        expect(h.finds.length).toBe(perCall);
+
+        // The bump this pin is about: never a local `insert`/`update`/`delete`
+        // on `h.engine` — exactly what a PEER's converged mutation looks like
+        // from this replica's own engine's point of view.
+        bumpWriteEpoch(h.engine, 'remote');
+
+        await h.protocol.getMetaItems({ type: 'object' });
+        expect(h.finds.length).toBe(perCall * 2);
+    });
+});
+
 // ═══════════════════════════════════════════════════════════════════════════
 // 4. Negative caching — the bulk of leg D's win (#11633 §1, §4)
 // ═══════════════════════════════════════════════════════════════════════════