test(metadata-protocol): re-measure the deleted-datasource prolongation across replicas after #13331's fix

[os-musk]

Part of #13609

Tests only. No source change: packages/metadata-protocol/src/protocol.ts, packages/metadata/src/metadata-manager.ts and packages/services/service-cluster/** are untouched, as the card's dispatch requires. The one non-test file is gate scaffolding (see "Gate union").

What this is

The maintainer's ruling (director batch A, 2026-09-01, 「同意。」) made #13609 the re-verification carrier for #13331's fix:

This card stays open as the re-verification carrier: when #13331's fix lands, re-measure the deleted-datasource prolongation on a multi-node deployment (the four candidate seams above give the checklist — listCache TTL is the one bounded residue that may legitimately remain, per #5109). Close only on that measurement.

#13331's fix landed (PR #14183, 1403d943). This PR is that measurement, shipped as a test rather than a comment. It is not a fix.

The discriminator is DURATION, and it is sharp: bounded (≲ one cache TTL window) = fixed; unbounded = not fixed. "It cleared eventually" is not a reading unless the bound is stated, so the harness does not assert a boolean — measureProlongationMs() returns a number of milliseconds, or null for unbounded, and the cases assert the number.

The harness, and what door it measures

packages/metadata-protocol/src/protocol.datasource-delete-prolongation.test.ts — one new file, beside protocol.cluster-mutation-fanout.test.ts, reusing its two-replica pub/sub shape.

Two protocol instances ("replicas") over ONE shared sys_metadata store, each with its OWN registry, its OWN overlay cache and its OWN write epoch, joined by a transport that delivers every publish to every subscriber (the publisher included — what a real remote driver does, and what the originNode loopback guard exists for). Replica A deletes through deleteMetaItem; replica B is measured.

The door under measurement is the protocol's, not the manager's. GET /api/v1/meta/:type is served by the REST list route whose body is const items = await p.getMetaItems(listRequest). That answer is built from two local, per-replica sources, which is what lets a peer serve a row the shared DB no longer has:

1. engine.registry.listItems(type) — the in-memory registry. It is the base of the answer; sys_metadata rows are merged on top (mergePackageAwareOverlay), and nothing in that merge prunes a base item whose row is gone. No TTL — the unbounded seam.
2. The meta-overlay-cache row-set cache — keyed on the engine's write epoch plus a TTL (OS_METADATA_OVERLAY_CACHE_TTL_MS, default 30s). The epoch is local, so a peer's write never moves it. The bounded seam.

Premise re-confirmed on this tree, not inherited: datasource is allowRuntimeCreate: true (packages/spec/src/kernel/metadata-plugin.zod.ts), so deleteMetaItem's useRepoPath fork takes the repository exit, which emits. The first case asserts exactly one metadata.mutated publish carrying {type: 'datasource', name, state: 'deleted', organizationId: null} — if the delete took a non-emitting exit, this re-verification would be measuring a channel the delete never uses.

The measurement

arm	kernel	bridge	read during residue window	prolongation
B	unscoped (control-plane)	attached	yes	⛔ UNBOUNDED — still served past 10 × 30s windows
B	unscoped (control-plane)	attached	no	BOUNDED — 30 000 ms (one window)
A (control)	unscoped	detached	either	⛔ UNBOUNDED — past 10 windows (the pre-fix shape)
B	scoped (env_prod)	attached	yes	BOUNDED — 30 000 ms (one window)

My pre-run prediction for Arm B was BOUNDED, and it was falsified — in the worse direction. The prediction's first half held and is pinned: the bridge really does heal the peer's registry, within one bus hop, before the clock moves at all. The second half did not:

A single read of the peer's own /api/v1/meta/datasource door, landing while the peer's stale overlay-cache entry is still fresh, writes the deleted row straight back into the registry the bridge just healed — and the registry has no TTL.

So the prolongation is neither unconditionally bounded nor unconditionally unbounded, and the discriminator is not time but traffic. Both halves are pinned side by side, differing in exactly that one step, because either alone reads as a clean verdict and neither alone is true.

On a replica actually serving /api/v1/meta/datasource — the door QA was watching — a read inside a 30s window is the ordinary case, not the unlucky one. The bounded arm is the quiet-replica arm.

Why the re-hydration happens

Nothing on the bridge's receipt path touches the write epoch that keys the overlay cache. applyRemoteMetadataMutation re-reads the row and repairs the registry; it performs no engine write, and a peer does no writing of its own on this path, so the peer's epoch does not move and its pre-delete row set stays "fresh" for the rest of its TTL. getMetaItems then does what the cache's own header says it always does, hit or miss — it runs hydrateOverlayIntoRegistry over those rows.

The comparison that makes this a gap rather than a design: the sibling bridge over the same substrate does bump it. packages/objectql/src/authz-invalidation-bridge.ts:71 calls epoch.bump('remote') when it applies a peer hint, and meta-overlay-cache.ts's own header cites that as the reason cross-node convergence "narrows for free" there. The metadata.mutated bridge added for #13331 contains no epoch reference at all. The two cross-node paths over one substrate disagree, and this door sits on the half that does not invalidate.

Not fixed here. The card is a measurement carrier and its source surface is read-only, so the reading is reported and the repair is left to a card that can be decided on it.

Controls

• Positive control (firing). Arm A reproduces the pre-fix unbounded prolongation, so the probe demonstrably can see one. The scoped case returns a finite number on the same instrument, so it can also see one end — the unbounded readings are measurements, not an instrument that never clears anything.
• Negative control. An unrelated datasource (analytics_db) is still served on the peer in all three shapes after 11 windows, so the readings are about the deleted row and not about a peer that stopped answering.
• Drain adequacy is asserted, not assumed. settle() drains microtasks without moving the clock; the registry-heal case asserts the heal did happen after exactly that drain, so an inadequate drain fails loudly there instead of silently inflating a prolongation elsewhere.

Four-seam checklist — a verdict on each

seam	verdict
1. pubsub / bridge not attached	RE-MEASURED. Arm A reproduces the unbounded pre-fix prolongation; the landed publisher + bridge do fan the DELETE out and do heal the peer's registry. This seam is CLOSED — and it is not the whole story, see 3.
2. restoreRuntimeDatasources re-seeding	ELIMINATED by PR #13883 — boot-only, reads the already-corrected DB. Cited, not re-derived.
3. list-cache TTL (#5109)	⛔ NOT the bounded residue the ruling expected. At this door the cache is meta-overlay-cache, not MetadataManager.listCache — and it does not merely delay the correct answer, it feeds the untimed registry, converting a 30s residue into an unbounded one on any read.
4. same class as #13578	ELIMINATED by PR #13883 — same symptom, opposite mechanism. Cited, not re-derived.

Neither the QA observation nor the source counter-evidence is discarded, and the file is written so it cannot be: Arm A and the read-during-window case reproduce the observed cluster-wide prolongation, and the heal case reproduces the counter-evidence that the delete path really does fan out. Both readings are true, then and now.

⛔ Live multi-node run: NOT MEASURED

No live cluster driver or multi-process deployment is reachable from the dev container (OS_TEST_* unset, no cluster driver beyond memory). The in-process two-instance harness with a real cross-instance transport is declared as the proxy, per the allowance in the card's own state-hygiene note; it is never presented as the live measurement.

What the proxy leaves open, named rather than papered over — the residual sub-question from PR #13883: whether the QA deployment was on the shipped in-process memory cluster driver (nothing listening) or hit a second defect in a genuinely distributed driver. Only a real-driver run separates those. Reported NOT MEASURED, not as a pass.

Ablation — direction predicted before the run

Predicted, in writing before running: flipping attach: true to false in the two bridge-dependent cases (the registry-heal case and the bounded no-read case), neutering the bridge in the harness, never in source, turns exactly those two RED while every other case stays GREEN. The already-unbounded cases cannot move — that is the point: they are unbounded with the bridge as well, so an ablation that reddened them would mean the harness, not the bridge, was deciding the outcome.

Measured — matched exactly: Tests 2 failed | 7 passed (9), the two failures being Arm B — WITH the landed publisher + bridge, the bridge DOES heal the peer registry and Arm B measured, door NOT read during the residue window: BOUNDED by one TTL window.

Mutation confirmed on disk before the run, anchored on both the injected and the removed text (an editor's exit code proves nothing — a zero-hit replace exits 0): injected=2 (expect 2), remaining_attach_true=3 (expect 3, was 5), blob 2a30331… becoming 00e7ee4…. No rebuild leg applies: the subject is imported as ./protocol.js, a relative specifier inside its own package that vitest resolves to src/protocol.ts, so nothing resolves through exports to dist/. Restore ran from an absolute-path trap … EXIT INT TERM pinned to HEAD (never a bare git checkout --, which restores from a possibly-polluted index) and was verified by state, not exit code: worktree blob back to 2a30331… = HEAD blob, git diff HEAD empty, zero ABLATED markers, clean git status.

Gate union

All of the below ran on the final commit 55b725689 (git rev-parse --short HEAD), with the family list re-derived on that same tree by node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands — the script's own change-set derivation, never a hand-rolled git diff, and with no STALE TREE warning. 34 families: 31 exit 0, 3 exit 3 (NOT MEASURED), 0 failures.

Exit codes captured after redirect (cmd > log 2>&1; ec=$?), never through a pipe.

• pnpm lint (the full repo scan, eslint . --no-inline-config): exit 0, zero problems. Run whole rather than narrowed, so no narrowing argument is owed.
• Package suite — pnpm --filter @objectstack/metadata-protocol test: Test Files 155 passed | 2 skipped (157), Tests 2134 passed | 10 skipped (2144).
• Package typecheck — pnpm --filter @objectstack/metadata-protocol typecheck: exit 0. The new file is in the program, measured rather than assumed: tsc --noEmit --listFiles | grep -c protocol.datasource-delete-prolongation.test.ts = 1.
• check:engine-double-contract — initially exit 1: the new file's delete/findOne/update doubles (all already routed through assertEngine*Dispatch) were not in the shrink-only ledger. Repaired the way the gate names, by node scripts/check-engine-double-contract.mjs --write and committing the three rows in scripts/engine-double-contract.pinned.json — never by raising a baseline. Re-run: exit 0. That commit is the one non-test file here, and it is why 7 further gate-script families joined the union; all 7 green.
• NOT MEASURED (exit 3, each the gate's own self-declared signal, recorded as neither pass nor fail): check-test-completeness.mjs ("PREREQUISITE NOT MET — this gate grades a saved turbo run test log, and no log was named"), check:dual-build-cjs-loads ("PREREQUISITE NOT MET — this gate reads built output, and some package has no dist/ … ⛔ This is NOT a pass: nothing was measured"), check:type-check-debt ("PREREQUISITE NOT MET"). All three need a full-farm build or a saved CI log that a local worktree does not produce; CI runs them on this PR.

---

🤖 Generated with Claude Code

https://claude.ai/code/session_0112hMx9hjJ9BgB28X97DS68

---

Generated by Claude Code

[github-actions]

📓 Docs Drift Check

Nothing in this diff resolved to a documentable surface (no symbol, route or SDK anchor derived from 0 changed package(s)), so this run has no opinion about the docs.

What this run could not see

• a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 0 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 96b627d135826253981cfa01d74b2832ccdea194 → packageMentionDocs.

[os-musk]

Provenance (domain:engine seat, session session_0112hMx9hjJ9BgB28X97DS68, 07:04Z): flipped to ready and auto-merge (squash) armed on head 55b725689. ACCEPT on the card: 5505575432 (#13609, Part of). Every check run on this head completed success or skipped (37 runs; Lint & Repo Gates 06:58:42Z, Test Core (1/6) 06:54:49Z); governed-surface test on the exact two-file list: NOT governed; skip-changeset on (tests + gate scaffolding). One reading recorded rather than hidden: the third-party Vercel commit status has sat at pending ("Vercel is deploying your app") since 06:34Z with no update — a preview deployment, not a check of the code; mergeable_state reads unstable for that reason and the queue's required checks are green. Landing to-do at MERGED: verify by content on origin/main, strip pm:dispatched from #13609 and add needs-user-decision in the same stroke (the four-facet block 5505583013 is already on the card; the card stays open on the maintainer's ruling), landing record.

---

Generated by Claude Code