From 0b6603fd250fe7736d36c0128ca2e2ace20f1936 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 2 Sep 2026 05:51:02 +0000 Subject: [PATCH 1/3] test(metadata-protocol): re-verify deleted-datasource prolongation across replicas MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit WIP — measurement harness for #13609: two protocol replicas over one shared sys_metadata store, measuring how long the peer's /api/v1/meta/datasource read door keeps serving a deleted datasource, with and without the landed #13331 publisher + cluster bridge. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_0112hMx9hjJ9BgB28X97DS68 --- ...col.datasource-delete-prolongation.test.ts | 587 ++++++++++++++++++ 1 file changed, 587 insertions(+) create mode 100644 packages/metadata-protocol/src/protocol.datasource-delete-prolongation.test.ts diff --git a/packages/metadata-protocol/src/protocol.datasource-delete-prolongation.test.ts b/packages/metadata-protocol/src/protocol.datasource-delete-prolongation.test.ts new file mode 100644 index 0000000000..0658a6daa7 --- /dev/null +++ b/packages/metadata-protocol/src/protocol.datasource-delete-prolongation.test.ts @@ -0,0 +1,587 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#13609] RE-VERIFICATION — how long does a PEER replica keep serving a + * DELETED datasource on `GET /api/v1/meta/datasource`? + * + * --------------------------------------------------------------------------- + * What this file is, and why it is a measurement rather than a fix + * --------------------------------------------------------------------------- + * QA observed a deleted datasource still being served by + * `/api/v1/meta/datasource` on all three replicas of a cluster, prolonging an + * exposure. The maintainer ruled (2026-09-01, director decision batch A) that + * the #13331 fix — a publisher at the protocol's mutation choke point, a + * `service-cluster` bridge, and peers re-reading their OWN DB — was expected to + * close this observation too, and that this card stays open as the + * re-verification carrier: re-measure the prolongation once that fix lands, and + * close only on the measurement. + * + * ⭐ The discriminator is DURATION, and it is sharp: + * + * bounded (≲ one cache TTL window) = fixed + * unbounded = not fixed + * + * "It cleared eventually" is not a reading unless the bound is stated. So this + * file does not assert a boolean; it MEASURES a bound (see + * {@link measureProlongationMs}) and asserts the number. + * + * --------------------------------------------------------------------------- + * The read door under measurement is the PROTOCOL's, not the manager's + * --------------------------------------------------------------------------- + * `GET /api/v1/meta/:type` is served by `rest-server.ts`' list route, whose + * body is `const items = await p.getMetaItems(listRequest)` — `p` being this + * protocol. So the door QA watched is `getMetaItems`, and its answer is built + * from TWO local, per-replica sources, which is what makes a peer able to serve + * a row the shared DB no longer has: + * + * 1. `engine.registry.listItems(type)` — the in-memory registry. It is the + * BASE of the answer, and `sys_metadata` rows are merged ON TOP of it + * (`mergePackageAwareOverlay`). Nothing in that merge prunes a base item + * whose row is gone. ⭐ The registry carries NO TTL, so a stale entry here + * is served until the process restarts — this is the UNBOUNDED seam, and + * it is the mechanism PR #13883 identified one layer up on + * `MetadataManager` (registry hit never re-checked against the loader). + * 2. The `meta-overlay-cache` row-set cache — keyed on the engine's write + * EPOCH plus a TTL (`OS_METADATA_OVERLAY_CACHE_TTL_MS`, default 30s). The + * epoch is LOCAL: a peer's write never moves it, so this cache's own + * header names exactly this residue — "a PEER node's write on a deployment + * with no bridge attached". ⭐ This is the BOUNDED seam. + * + * ⚠️ A precision the card's checklist did not have: at THIS door the bounded + * residue the ruling allows ("listCache TTL is the one bounded residue that may + * legitimately remain, per #5109") is `meta-overlay-cache`'s TTL, not + * `MetadataManager.LIST_CACHE_TTL_MS`. Both default to 30_000 ms, so the + * ruling's bound is unchanged in magnitude — but the cache that holds the + * residue is a different one, and a future author tuning `LIST_CACHE_TTL_MS` + * would not move this number. The constant is imported here, never retyped, so + * this pin tracks whichever value ships. + * + * --------------------------------------------------------------------------- + * Topology: a PROXY for a live multi-node deployment, declared as one + * --------------------------------------------------------------------------- + * ⛔ This is NOT a live multi-node run, and must never be read as one. No live + * cluster driver or multi-process deployment is reachable from the dev + * container, so the live measurement the ruling asks for is reported NOT + * MEASURED. What runs here is the shape PR #13331's own fan-out pin uses and + * that the PM declared acceptable as the proxy: TWO protocol instances over ONE + * shared `sys_metadata` store, each with its OWN registry, its OWN overlay + * cache and its OWN write epoch, joined by a real cross-instance transport that + * delivers every publish to every subscriber (the publisher included — what a + * real remote driver does, and what the `originNode` loopback guard exists + * for). + * + * What the proxy leaves open, named rather than papered over: whether the QA + * deployment was running the shipped in-process `memory` cluster driver (which + * has no cross-process delivery, so nothing was listening) or hit a SECOND + * defect in a genuinely distributed driver. Only a real-driver run separates + * those, and it is not run here. + * + * --------------------------------------------------------------------------- + * Arms, with directions declared BEFORE the run + * --------------------------------------------------------------------------- + * • Arm B (bridge attached — the landed #13331 shape): the peer's registry + * converges on the DELETE within one bus hop, so the unbounded seam is shut; + * the peer's own overlay cache still serves the pre-delete row set until its + * TTL lapses. -> BOUNDED at exactly one TTL window + * • Arm A (control, no bridge — the pre-fix production shape): the peer's + * registry is never healed, so the entry outlives the cache TTL and every + * window after it. -> UNBOUNDED (still served past 10 windows) + * • Negative control: an unrelated datasource on the peer is untouched by + * either arm. -> still served in both + * + * Arm A is what makes Arm B a measurement: it proves this probe CAN see a + * prolongation, so Arm B's bound is the bridge's doing and not an artifact of a + * harness that forgets rows on its own. + * + * --------------------------------------------------------------------------- + * Four-seam checklist (the card's own elimination list) — verdicts + * --------------------------------------------------------------------------- + * 1. pubsub / bridge not attached ....... RE-MEASURED here. Arm A reproduces + * the unbounded prolongation; Arm B shows the landed publisher + bridge + * bounds it. This is the seam, and it is now closed to a TTL. + * 2. `restoreRuntimeDatasources` re-seeding ... ELIMINATED by PR #13883 — + * boot-only, and it reads the already-corrected DB, so it cannot explain a + * steady-state cross-replica prolongation without a restart. Not re-derived + * here (the card's dispatch forbids re-deriving §A). + * 3. list-cache TTL (#5109) ............. The one BOUNDED residue the ruling + * allows, and it is what Arm B measures — refined to the overlay cache, see + * the precision note above. + * 4. same class as #13578 ............... ELIMINATED by PR #13883 — same + * symptom, opposite mechanism (that driver registry had NO eviction door; + * this one's door exists and does broadcast). Not re-derived here. + * + * ⛔ Neither the QA observation nor the source counter-evidence is discarded by + * this file, and it is written so it cannot be: Arm A reproduces the observed + * cluster-wide prolongation, and Arm B reproduces the counter-evidence that the + * delete path really does fan out. Both readings are true; the seam was the + * transport between them. + * + * --------------------------------------------------------------------------- + * Reverse verification (ablation), direction predicted BEFORE the run + * --------------------------------------------------------------------------- + * Flipping Arm B's `attach: true` to `false` — neutering the bridge in the + * HARNESS, never in source, which this card may not touch — must turn EXACTLY + * the Arm-B bound case RED, reporting an unbounded prolongation where a + * one-window bound is expected, while Arm A, the negative control and the + * mechanism cases stay GREEN. No rebuild leg applies: this test imports the + * subject as `./protocol.js`, a relative specifier inside its own package that + * vitest resolves to `src/protocol.ts`, so nothing is served from `dist/`. + * Measured in the PR body. + */ + +import { beforeEach, afterEach, describe, expect, it, vi } from 'vitest'; +import { assertEngineDeleteDispatch, assertEngineUpdateDispatch, assertEngineFindOnePredicate } from '@objectstack/metadata-core'; +import { META_OVERLAY_CACHE_DEFAULT_TTL_MS } from './meta-overlay-cache.js'; +import { + ObjectStackProtocolImplementation, + METADATA_MUTATION_CLUSTER_CHANNEL, + type ClusterMetadataMutationPayload, +} from './protocol.js'; + +/** The residue window this door is bounded by. Imported, never retyped. */ +const TTL_MS = META_OVERLAY_CACHE_DEFAULT_TTL_MS; + +/** How far past the bound Arm A is chased before it is called unbounded. */ +const WINDOWS_PROBED = 10; + +interface Row { + id: string; + type: string; + name: string; + organization_id: string | null; + package_id: string | null; + state: string; + metadata: string; + checksum?: string; + version?: number; +} + +interface HistoryRow { + id: string; + type: string; + name: string; + version: number; + organization_id: string | null; + operation_type: string; + metadata?: string | null; + recorded_at?: string; +} + +/** ADR-0048 overlay key — (type, name, org, state, package_id). */ +const keyOf = (w: Record) => + `${w.type}|${w.name}|${w.organization_id ?? '__env__'}|${w.state ?? 'active'}|${w.package_id ?? '__nopkg__'}`; + +function matchesWhere(r: Row, where: Record): boolean { + for (const [k, v] of Object.entries(where)) { + if (k === '$or') { + const clauses = v as Array>; + if (!clauses.some((c) => matchesWhere(r, c))) return false; + continue; + } + if (v === undefined) continue; + if ((r as unknown as Record)[k] !== v) return false; + } + return true; +} + +/** The SHARED database — the one thing every replica of a deployment agrees on. */ +function makeSharedStore() { + return { + rows: new Map(), + historyRows: [] as HistoryRow[], + nextId: { value: 0 }, + }; +} + +/** + * A registry that actually STORES what is registered, because the answer under + * measurement is what `listItems` returns — a call-recording double would make + * the read door answer `[]` on every replica and the prolongation would be + * invisible by construction. + * + * `removeRuntimeShadow` answers `false` (no packaged artifact underneath) and + * no `metadata` service is installed, so `restoreArtifactRegistryView`'s tier + * walk reaches tier 3 — `removeOverlayEntry` — which is the limb that retires a + * runtime-only row. That is the shape a runtime-authored datasource has: the + * row WAS the item, with no code-shipped layer under it. + */ +function makeRegistry() { + const byType = new Map>(); + const removedEntries: string[] = []; + const bucket = (type: string) => { + let m = byType.get(type); + if (!m) byType.set(type, (m = new Map())); + return m; + }; + return { + removedEntries, + registry: { + registerItem: (type: string, item: any) => { + if (item && typeof item === 'object' && typeof item.name === 'string') { + bucket(type).set(item.name, item); + } + }, + listItems: (type: string) => Array.from(byType.get(type)?.values() ?? []), + getItem: (type: string, name: string) => byType.get(type)?.get(name), + removeOverlayEntry: (type: string, name: string) => { + removedEntries.push(`${type}|${name}`); + return bucket(type).delete(name); + }, + removeRuntimeShadow: () => false, + registerObject: () => {}, + unregisterObject: () => true, + removeObjectOverlay: () => {}, + getObject: () => undefined, + getPackage: () => undefined, + getArtifactItem: () => undefined, + }, + }; +} + +/** + * One "replica": a stub engine over the SHARED store, with its OWN registry and + * its OWN write-epoch seam, wrapped by its own protocol instance. + * + * ⭐ The write epoch is REQUIRED for this measurement, not decoration. The + * overlay cache declines outright on an engine that exposes no epoch seam + * ("only a real engine caches"), so a double without one would make the bounded + * residue unobservable and Arm B would report a zero-length prolongation that + * no shipped deployment has. Modelling the seam is what puts the residue on the + * clock. + * + * The kernel is UNSCOPED (`environmentId === undefined`) — the control-plane + * shape. That is the shape in which `getMetaItems` hydrates overlay rows into + * the registry, which is how a replica that merely SERVED the datasource comes + * to hold it locally. That is the QA-observed precondition: all three replicas + * were serving the entry before the delete. + */ +function makeReplica(store: ReturnType) { + const { rows, historyRows, nextId } = store; + const { registry, removedEntries } = makeRegistry(); + + const findRow = (w: Record): { key: string; row: Row } | null => { + if (w.id !== undefined) { + for (const [k, r] of rows) if (r.id === w.id) return { key: k, row: r }; + return null; + } + if (w.package_id !== undefined) { + const k = keyOf(w); + const r = rows.get(k); + if (r) return { key: k, row: r }; + } + for (const [k, r] of rows) if (matchesWhere(r, w)) return { key: k, row: r }; + return null; + }; + + const matchesHistory = (h: HistoryRow, w: Record): boolean => { + if (w.organization_id !== undefined && h.organization_id !== w.organization_id) return false; + if (w.type !== undefined && h.type !== w.type) return false; + if (w.name !== undefined && h.name !== w.name) return false; + if (w.version !== undefined && h.version !== w.version) return false; + if (w.operation_type !== undefined && h.operation_type !== w.operation_type) return false; + return true; + }; + + // The write-epoch seam, per replica. Any write advances it; a PEER's write + // does not — which is the whole reason the overlay cache has a TTL at all. + const writeEpoch = { + current: 0, + bump() { this.current += 1; }, + subscribe: () => () => {}, + }; + + const engine: any = { + writeEpoch, + async findOne(table: string, opts: { where: Record }) { + assertEngineFindOnePredicate(table, opts); + if (table === 'sys_metadata_history') { + return historyRows.find((h) => matchesHistory(h, opts.where)) ?? null; + } + if (table !== 'sys_metadata') return null; + return findRow(opts.where)?.row ?? null; + }, + async find(table: string, opts?: { where?: Record; limit?: number }) { + // Honour the caller's bound AFTER the filter, by PRESENCE — the + // objectql-double-limit contract. + const bound = (all: T[]): T[] => + typeof opts?.limit === 'number' ? all.slice(0, opts.limit) : all; + if (table === 'sys_metadata_history') { + return bound(historyRows.filter((h) => matchesHistory(h, opts?.where ?? {}))); + } + if (table !== 'sys_metadata') return []; + return bound(Array.from(rows.values()).filter((r) => matchesWhere(r, opts?.where ?? {}))); + }, + async insert(table: string, data: Record) { + if (table === 'sys_metadata_audit') return { id: 'audit_skip' }; + if (table === 'sys_metadata_history') { + nextId.value += 1; + const h = { ...(data as unknown as HistoryRow), id: `h_${nextId.value}` }; + historyRows.push(h); + return { id: h.id }; + } + if (table !== 'sys_metadata') return { id: 'side_effect_skip' }; + nextId.value += 1; + const row = { ...(data as unknown as Row), id: `r_${nextId.value}` }; + rows.set(keyOf(data), row); + writeEpoch.bump(); + return { id: row.id }; + }, + async update(_t: string, data: Record, opts: { where: Record }) { + assertEngineUpdateDispatch(data, opts); + const found = findRow(opts.where); + if (!found) return { id: null }; + const merged = { ...found.row, ...(data as unknown as Row) }; + rows.delete(found.key); + rows.set(keyOf(merged), merged); + writeEpoch.bump(); + return { id: found.row.id }; + }, + async delete(_t: string, opts: { where: Record }) { + assertEngineDeleteDispatch(opts); + const found = findRow(opts.where); + if (!found) return { deleted: 0 }; + rows.delete(found.key); + writeEpoch.bump(); + return { deleted: 1 }; + }, + async count() { return 0; }, + async transaction(cb: (ctx: unknown, info: { owned: boolean }) => Promise): Promise { + return cb(undefined, { owned: true }); + }, + async syncObjectSchema() { return true; }, + async dropObjectSchema() { return true; }, + registry, + }; + + const protocol = new ObjectStackProtocolImplementation( + engine, () => new Map(), undefined, + ) as any; + + return { protocol, engine, registry, removedEntries }; +} + +/** + * A remote-driver-shaped bus: one transport, every publish delivered to EVERY + * subscription — the publisher's own node included. + */ +function makeBus() { + const subs: Array<{ channel: string; handler: (msg: { channel: string; payload: unknown; publishedAt: number }) => void }> = []; + const published: Array<{ channel: string; payload: ClusterMetadataMutationPayload }> = []; + const bus = { + async publish(channel: string, payload: unknown) { + published.push({ channel, payload: payload as ClusterMetadataMutationPayload }); + for (const s of [...subs]) { + if (s.channel === channel) s.handler({ channel, payload, publishedAt: Date.now() }); + } + }, + subscribe(channel: string, handler: (msg: never) => void) { + const sub = { channel, handler: handler as (msg: { channel: string; payload: unknown; publishedAt: number }) => void }; + subs.push(sub); + return () => { + const i = subs.indexOf(sub); + if (i >= 0) subs.splice(i, 1); + }; + }, + async close() {}, + }; + return { bus, published }; +} + +/** A runtime-authored datasource body — the shape a `PUT /meta/datasource/:name` writes. */ +const datasourceBody = (name: string) => ({ + name, + label: `Datasource ${name}`, + driver: 'postgres', + config: { host: 'db.internal', database: name }, +}); + +/** Both replicas over one store, joined (or not) by the bus. */ +function makeCluster(opts: { attach: boolean }) { + const store = makeSharedStore(); + const writer = makeReplica(store); + const peer = makeReplica(store); + const { bus, published } = makeBus(); + if (opts.attach) { + writer.protocol.attachMetadataMutationPubSub(bus, 'node-a'); + peer.protocol.attachMetadataMutationPubSub(bus, 'node-b'); + } + return { store, writer, peer, bus, published }; +} + +/** + * Settle the cluster without moving the clock. + * + * The bridge's receipt path (`applyRemoteMetadataMutation`) is async and + * fire-and-forget — `deleteMetaItem` returns before the peer has converged — + * but it awaits only PROMISES, never timers. So draining microtasks is the + * correct and complete settle, and it leaves the fake clock untouched, which + * matters because the clock is the instrument here. + * + * ⭐ Its adequacy is not assumed: the Arm-B convergence case below asserts the + * peer's registry IS healed after exactly this drain, so an insufficient drain + * fails loudly there rather than silently inflating a prolongation elsewhere. + */ +async function settle(): Promise { + for (let i = 0; i < 50; i++) await Promise.resolve(); +} + +/** What `GET /api/v1/meta/datasource` answers on this replica, by name. */ +async function servedNames(replica: { protocol: any }): Promise { + const items = await replica.protocol.getMetaItems({ type: 'datasource' }); + return (items as Array<{ name?: unknown }>).map((i) => String(i?.name)); +} + +const serves = async (replica: { protocol: any }, name: string): Promise => + (await servedNames(replica)).includes(name); + +/** + * ⭐ THE MEASUREMENT. Advance the peer's clock one TTL window at a time and + * return how long it kept serving `name`, in ms — or `null` for UNBOUNDED, + * meaning it was still serving the deleted entry after {@link WINDOWS_PROBED} + * full windows. + * + * Returns a DURATION rather than a boolean on purpose: the card's pass/fail is + * a bound, and "it cleared eventually" is not a reading unless the number is + * stated. + */ +async function measureProlongationMs( + replica: { protocol: any }, + name: string, +): Promise { + if (!(await serves(replica, name))) return 0; + for (let window = 1; window <= WINDOWS_PROBED; window++) { + vi.setSystemTime(Date.now() + TTL_MS); + await settle(); + if (!(await serves(replica, name))) return window * TTL_MS; + } + return null; +} + +/** + * Bring both replicas to the QA-observed pre-condition: a runtime-authored + * datasource that BOTH replicas are serving on their own read door. + * + * The peer's read is what hydrates the row into the peer's registry — the local + * copy that later outlives the shared row. Doing it through the door (rather + * than by reaching into the registry) is the point: this is exactly how a + * replica that only ever SERVED the entry ends up holding it. + */ +async function seedServedDatasource( + cluster: ReturnType, + name: string, +): Promise { + const res = await cluster.writer.protocol.saveMetaItem({ + type: 'datasource', name, item: datasourceBody(name), mode: 'publish', + }); + expect(res.success).toBe(true); + await settle(); + expect(await serves(cluster.writer, name)).toBe(true); + expect(await serves(cluster.peer, name)).toBe(true); +} + +describe('[#13609] the read door under measurement', () => { + it('`datasource` takes the EMITTING repository delete exit — the premise this card rests on', async () => { + const { writer, published } = makeCluster({ attach: true }); + await writer.protocol.saveMetaItem({ + type: 'datasource', name: 'billing_db', item: datasourceBody('billing_db'), mode: 'publish', + }); + await settle(); + published.length = 0; + + const res = await writer.protocol.deleteMetaItem({ type: 'datasource', name: 'billing_db' }); + expect(res.success).toBe(true); + await settle(); + + // `datasource` is `allowRuntimeCreate: true`, so `deleteMetaItem`'s + // `useRepoPath` fork takes the REPOSITORY exit — the one that emits. + // A code-only exit would publish nothing and this card's + // re-verification would be measuring a channel the delete never uses. + expect(published).toHaveLength(1); + expect(published[0].channel).toBe(METADATA_MUTATION_CLUSTER_CHANNEL); + expect(published[0].payload.event).toEqual({ + type: 'datasource', name: 'billing_db', state: 'deleted', organizationId: null, + }); + }); + + it('the peer serves the row from its OWN registry, so the shared DB alone cannot answer for it', async () => { + const cluster = makeCluster({ attach: false }); + await seedServedDatasource(cluster, 'billing_db'); + + // The peer never wrote this row; it hydrated it by READING. This is the + // local copy whose lifetime the rest of the file measures. + expect(cluster.peer.registry.listItems('datasource').map((i: any) => i.name)) + .toEqual(['billing_db']); + }); +}); + +describe('[#13609] ⭐ the re-verification: how long does the peer keep serving a DELETED datasource?', () => { + beforeEach(() => { vi.useFakeTimers(); }); + afterEach(() => { vi.useRealTimers(); }); + + it('Arm B — WITH the landed publisher + bridge: BOUNDED at exactly one overlay-cache TTL window', async () => { + const cluster = makeCluster({ attach: true }); + await seedServedDatasource(cluster, 'billing_db'); + + const res = await cluster.writer.protocol.deleteMetaItem({ type: 'datasource', name: 'billing_db' }); + expect(res.success).toBe(true); + await settle(); + + // ⭐ The unbounded seam is SHUT: the peer's registry — the copy with no + // TTL — is healed within one bus hop, before the clock moves at all. + // This assertion is also what proves `settle()` is a sufficient drain. + expect(cluster.peer.registry.listItems('datasource')).toEqual([]); + expect(cluster.peer.removedEntries).toContain('datasource|billing_db'); + + // …and what remains is the bounded residue, on the clock. + const prolongation = await measureProlongationMs(cluster.peer, 'billing_db'); + expect(prolongation).not.toBeNull(); + expect(prolongation).toBe(TTL_MS); + + // Stated as the ruling asks for it: a NUMBER, and a bound. + expect(TTL_MS).toBe(30_000); + }); + + it('Arm A — CONTROL, no bridge (the pre-fix shape): UNBOUNDED, still served past 10 windows', async () => { + const cluster = makeCluster({ attach: false }); + await seedServedDatasource(cluster, 'billing_db'); + + const res = await cluster.writer.protocol.deleteMetaItem({ type: 'datasource', name: 'billing_db' }); + expect(res.success).toBe(true); + await settle(); + + // The writer's own door is correct immediately — the asymmetry QA saw. + expect(await serves(cluster.writer, 'billing_db')).toBe(false); + + // The peer's registry was never healed: no transport carried the signal. + expect(cluster.peer.registry.listItems('datasource').map((i: any) => i.name)) + .toEqual(['billing_db']); + + // ⭐ This is the firing positive control. It proves the probe above CAN + // see a prolongation — Arm B's bound is the bridge's doing, not a + // harness that drops rows by itself. + const prolongation = await measureProlongationMs(cluster.peer, 'billing_db'); + expect(prolongation).toBeNull(); + + // And it outlives the cache window by a factor of ten, which is why the + // original QA prolongation read LONGER than any TTL: the stale answer + // is not in anything that expires. + expect(await serves(cluster.peer, 'billing_db')).toBe(true); + }); + + it('negative control — an unrelated datasource is untouched by either arm', async () => { + for (const attach of [true, false]) { + const cluster = makeCluster({ attach }); + await seedServedDatasource(cluster, 'billing_db'); + await seedServedDatasource(cluster, 'analytics_db'); + + await cluster.writer.protocol.deleteMetaItem({ type: 'datasource', name: 'billing_db' }); + await settle(); + vi.setSystemTime(Date.now() + TTL_MS * (WINDOWS_PROBED + 1)); + await settle(); + + // The delete reached exactly one name on the peer, whichever arm. + expect(await serves(cluster.peer, 'analytics_db')).toBe(true); + expect(await serves(cluster.peer, 'billing_db')).toBe(attach ? false : true); + } + }); +}); From fd4fe4171b0fce30a99ed24792b0994e50a22b7f Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 2 Sep 2026 06:10:18 +0000 Subject: [PATCH 2/3] test(metadata-protocol): measure deleted-datasource prolongation across replicas Re-verification for #13609, the carrier the maintainer's ruling left open for it: with #13331's publisher + service-cluster bridge landed, how long does a peer replica keep serving a deleted datasource on the /api/v1/meta/datasource read door? Two protocol replicas over one shared sys_metadata store, each with its own registry, overlay cache and write epoch, joined by a real cross-instance transport. Declared as a PROXY for a live multi-node deployment, which is not reachable from this container. Measured: the bridge does heal the peer's registry within one bus hop, but the peer's overlay cache is not invalidated by the convergence, and any read of the peer's own door inside that residue window re-hydrates the deleted row back into the untimed registry. So the prolongation is bounded by one 30s TTL window only when no read lands in the window, and unbounded when one does. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_0112hMx9hjJ9BgB28X97DS68 --- ...col.datasource-delete-prolongation.test.ts | 359 +++++++++++++----- 1 file changed, 267 insertions(+), 92 deletions(-) diff --git a/packages/metadata-protocol/src/protocol.datasource-delete-prolongation.test.ts b/packages/metadata-protocol/src/protocol.datasource-delete-prolongation.test.ts index 0658a6daa7..2a30331178 100644 --- a/packages/metadata-protocol/src/protocol.datasource-delete-prolongation.test.ts +++ b/packages/metadata-protocol/src/protocol.datasource-delete-prolongation.test.ts @@ -47,14 +47,12 @@ * header names exactly this residue — "a PEER node's write on a deployment * with no bridge attached". ⭐ This is the BOUNDED seam. * - * ⚠️ A precision the card's checklist did not have: at THIS door the bounded - * residue the ruling allows ("listCache TTL is the one bounded residue that may - * legitimately remain, per #5109") is `meta-overlay-cache`'s TTL, not - * `MetadataManager.LIST_CACHE_TTL_MS`. Both default to 30_000 ms, so the - * ruling's bound is unchanged in magnitude — but the cache that holds the - * residue is a different one, and a future author tuning `LIST_CACHE_TTL_MS` - * would not move this number. The constant is imported here, never retyped, so - * this pin tracks whichever value ships. + * ⚠️ A precision the card's checklist did not have: at THIS door the cache in + * play is `meta-overlay-cache`'s, not `MetadataManager.LIST_CACHE_TTL_MS`. Both + * default to 30_000 ms, so the ruling's bound is unchanged in magnitude — but + * they are different caches, a future author tuning `LIST_CACHE_TTL_MS` would + * not move this number, and only this one feeds the registry. The constant is + * imported here, never retyped, so this pin tracks whichever value ships. * * --------------------------------------------------------------------------- * Topology: a PROXY for a live multi-node deployment, declared as one @@ -77,53 +75,101 @@ * those, and it is not run here. * * --------------------------------------------------------------------------- - * Arms, with directions declared BEFORE the run + * Arms — predicted BEFORE the run, and the prediction was FALSIFIED * --------------------------------------------------------------------------- + * Predicted, in writing, before running: + * * • Arm B (bridge attached — the landed #13331 shape): the peer's registry - * converges on the DELETE within one bus hop, so the unbounded seam is shut; - * the peer's own overlay cache still serves the pre-delete row set until its - * TTL lapses. -> BOUNDED at exactly one TTL window - * • Arm A (control, no bridge — the pre-fix production shape): the peer's - * registry is never healed, so the entry outlives the cache TTL and every - * window after it. -> UNBOUNDED (still served past 10 windows) - * • Negative control: an unrelated datasource on the peer is untouched by - * either arm. -> still served in both - * - * Arm A is what makes Arm B a measurement: it proves this probe CAN see a - * prolongation, so Arm B's bound is the bridge's doing and not an artifact of a - * harness that forgets rows on its own. + * converges within one bus hop, so the unbounded seam is shut and the only + * residue is the peer's own overlay cache. -> BOUNDED at one TTL window + * • Arm A (control, no bridge — the pre-fix shape): the peer's registry is + * never healed. -> UNBOUNDED past 10 windows + * • Negative control: an unrelated datasource is untouched in every arm. + * + * ⭐ MEASURED: Arm A came out as predicted. Arm B did NOT, and the direction of + * the miss is the finding — it is worse than predicted, not better. The first + * half of the prediction holds: the bridge really does heal the peer's registry + * within one bus hop, before the clock moves (case 3 below pins it). The second + * half does not: + * + * A SINGLE READ of the peer's own `/api/v1/meta/datasource` door, + * landing while the peer's stale overlay-cache entry is still fresh, + * writes the deleted row straight back into the registry the bridge + * just healed — and the registry has no TTL. + * + * So the prolongation is neither unconditionally bounded nor unconditionally + * unbounded, and the discriminator is not time but TRAFFIC: + * + * read lands inside the residue window -> UNBOUNDED (cases 4, 6) + * no read lands inside it -> BOUNDED, one TTL window (case 5) + * + * Both are pinned, side by side and differing in exactly that one step, because + * either one alone reads as a clean verdict and neither one alone is true. + * + * ⚠️ On a replica actually serving `/api/v1/meta/datasource` — the door QA was + * watching — a read inside a 30s window is the ordinary case, not the unlucky + * one. The bounded arm is the quiet-replica arm. + * + * --------------------------------------------------------------------------- + * Why the re-hydration happens, precisely + * --------------------------------------------------------------------------- + * Nothing on the bridge's receipt path touches the WRITE EPOCH that keys the + * overlay cache. `applyRemoteMetadataMutation` re-reads the row and repairs the + * registry; it performs no engine write, and a peer replica does no writing of + * its own on this path, so the peer's epoch does not move and its pre-delete + * row set stays "fresh" for the rest of its TTL. `getMetaItems` then does what + * the cache's own header says it always does, hit or miss — it runs + * `hydrateOverlayIntoRegistry` over those rows. + * + * ⭐ The comparison that makes this a gap rather than a design: the SIBLING + * bridge for the same substrate does bump it. `authz-invalidation-bridge.ts` + * calls `epoch.bump('remote')` when it applies a peer hint, and the overlay + * cache's own header cites that as the reason cross-node convergence "narrows + * for free" there. The `metadata.mutated` bridge added for #13331 contains no + * epoch reference at all. So the two cross-node paths over one substrate + * disagree, and this door sits on the half that does not invalidate. + * + * ⛔ NOT FIXED HERE. This card is a measurement carrier and its source surface + * is read-only, so the reading is reported and the repair is left to a card + * that can be decided on it. * * --------------------------------------------------------------------------- * Four-seam checklist (the card's own elimination list) — verdicts * --------------------------------------------------------------------------- - * 1. pubsub / bridge not attached ....... RE-MEASURED here. Arm A reproduces - * the unbounded prolongation; Arm B shows the landed publisher + bridge - * bounds it. This is the seam, and it is now closed to a TTL. + * 1. pubsub / bridge not attached ....... RE-MEASURED. Arm A reproduces the + * unbounded pre-fix prolongation; the landed publisher + bridge do fan the + * DELETE out and do heal the peer's registry. This seam is CLOSED — and it + * is not the whole story, see 3. * 2. `restoreRuntimeDatasources` re-seeding ... ELIMINATED by PR #13883 — * boot-only, and it reads the already-corrected DB, so it cannot explain a - * steady-state cross-replica prolongation without a restart. Not re-derived - * here (the card's dispatch forbids re-deriving §A). - * 3. list-cache TTL (#5109) ............. The one BOUNDED residue the ruling - * allows, and it is what Arm B measures — refined to the overlay cache, see - * the precision note above. + * steady-state cross-replica prolongation without a restart. Cited, not + * re-derived (the dispatch forbids re-deriving that finding). + * 3. list-cache TTL (#5109) ............. ⛔ NOT the bounded residue the ruling + * expected. At this door the cache is `meta-overlay-cache`, not + * `MetadataManager.listCache` — and it does not merely delay the correct + * answer, it FEEDS the untimed registry, converting a 30s residue into an + * unbounded one on any read. That conversion is what this file measures. * 4. same class as #13578 ............... ELIMINATED by PR #13883 — same * symptom, opposite mechanism (that driver registry had NO eviction door; - * this one's door exists and does broadcast). Not re-derived here. + * this one's door exists and does broadcast). Cited, not re-derived. * * ⛔ Neither the QA observation nor the source counter-evidence is discarded by - * this file, and it is written so it cannot be: Arm A reproduces the observed - * cluster-wide prolongation, and Arm B reproduces the counter-evidence that the - * delete path really does fan out. Both readings are true; the seam was the - * transport between them. + * this file, and it is written so it cannot be: Arm A and case 4 reproduce the + * observed cluster-wide prolongation, and case 3 reproduces the counter- + * evidence that the delete path really does fan out and really does heal. Both + * readings are true, then and now. * * --------------------------------------------------------------------------- * Reverse verification (ablation), direction predicted BEFORE the run * --------------------------------------------------------------------------- - * Flipping Arm B's `attach: true` to `false` — neutering the bridge in the - * HARNESS, never in source, which this card may not touch — must turn EXACTLY - * the Arm-B bound case RED, reporting an unbounded prolongation where a - * one-window bound is expected, while Arm A, the negative control and the - * mechanism cases stay GREEN. No rebuild leg applies: this test imports the + * Flipping the `attach: true` of the two cases that depend on the bridge — the + * registry-heal case and the bounded no-read case — to `false`, neutering the + * bridge in the HARNESS and never in source, which this card may not touch, + * must turn EXACTLY those two RED (an unhealed registry, and a row still served + * after the window) while every other case stays GREEN. The already-unbounded + * cases cannot move, which is the point: they are unbounded with the bridge as + * well, so an ablation that reddened them would mean the harness, not the + * bridge, was deciding the outcome. No rebuild leg applies: this test imports the * subject as `./protocol.js`, a relative specifier inside its own package that * vitest resolves to `src/protocol.ts`, so nothing is served from `dist/`. * Measured in the PR body. @@ -228,6 +274,11 @@ function makeRegistry() { return bucket(type).delete(name); }, removeRuntimeShadow: () => false, + // The read door filters by package state and merges nav + // contributions for `app`; neither is under measurement here, so + // both answer the neutral value rather than being left absent. + isPackageDisabled: () => false, + applyNavContributions: (app: unknown) => app, registerObject: () => {}, unregisterObject: () => true, removeObjectOverlay: () => {}, @@ -255,7 +306,7 @@ function makeRegistry() { * to hold it locally. That is the QA-observed precondition: all three replicas * were serving the entry before the delete. */ -function makeReplica(store: ReturnType) { +function makeReplica(store: ReturnType, environmentId?: string) { const { rows, historyRows, nextId } = store; const { registry, removedEntries } = makeRegistry(); @@ -354,7 +405,7 @@ function makeReplica(store: ReturnType) { }; const protocol = new ObjectStackProtocolImplementation( - engine, () => new Map(), undefined, + engine, () => new Map(), environmentId, ) as any; return { protocol, engine, registry, removedEntries }; @@ -395,11 +446,20 @@ const datasourceBody = (name: string) => ({ config: { host: 'db.internal', database: name }, }); -/** Both replicas over one store, joined (or not) by the bus. */ -function makeCluster(opts: { attach: boolean }) { +/** + * Both replicas over one store, joined (or not) by the bus. + * + * `environmentId` selects the KERNEL SHAPE, and it is a variable of this + * measurement rather than a detail: it is the flag `getMetaItems` and + * `applyRegistryWriteThrough` both gate registry hydration on, so it decides + * whether a replica keeps a local registry copy of an overlay row at all — + * which is precisely the difference between a bounded and an unbounded + * prolongation. Unscoped (`undefined`) is the control-plane shape. + */ +function makeCluster(opts: { attach: boolean; environmentId?: string }) { const store = makeSharedStore(); - const writer = makeReplica(store); - const peer = makeReplica(store); + const writer = makeReplica(store, opts.environmentId); + const peer = makeReplica(store, opts.environmentId); const { bus, published } = makeBus(); if (opts.attach) { writer.protocol.attachMetadataMutationPubSub(bus, 'node-a'); @@ -425,10 +485,21 @@ async function settle(): Promise { for (let i = 0; i < 50; i++) await Promise.resolve(); } -/** What `GET /api/v1/meta/datasource` answers on this replica, by name. */ +/** + * What `GET /api/v1/meta/datasource` answers on this replica, by name. + * + * Both response shapes are unwrapped, exactly as the REST list route unwraps + * them: `getMetaItems` is typed `{ type, items[] }` while the implementation + * may answer the bare array, and the route handles both. Reading only one shape + * here would make the door's answer look empty and every prolongation below + * would measure zero. + */ async function servedNames(replica: { protocol: any }): Promise { - const items = await replica.protocol.getMetaItems({ type: 'datasource' }); - return (items as Array<{ name?: unknown }>).map((i) => String(i?.name)); + const answer = await replica.protocol.getMetaItems({ type: 'datasource' }); + const items: Array<{ name?: unknown }> = Array.isArray(answer) + ? answer + : ((answer as { items?: Array<{ name?: unknown }> })?.items ?? []); + return items.map((i) => String(i?.name)); } const serves = async (replica: { protocol: any }, name: string): Promise => @@ -458,25 +529,37 @@ async function measureProlongationMs( } /** - * Bring both replicas to the QA-observed pre-condition: a runtime-authored - * datasource that BOTH replicas are serving on their own read door. + * Bring both replicas to the QA-observed pre-condition: runtime-authored + * datasources that BOTH replicas are serving on their own read door. + * + * Every name is written BEFORE the peer's first read, deliberately. The peer's + * overlay cache memoises the whole row SET for `datasource`, and the peer's own + * write epoch never moves (it does no writing), so a name created after the + * peer has read once is invisible to the peer until that cache lapses. Seeding + * name-by-name through the peer's door would therefore fail on the second name + * for a reason that has nothing to do with deletes. * - * The peer's read is what hydrates the row into the peer's registry — the local - * copy that later outlives the shared row. Doing it through the door (rather - * than by reaching into the registry) is the point: this is exactly how a - * replica that only ever SERVED the entry ends up holding it. + * The peer's read is what hydrates the rows into the peer's registry on an + * unscoped kernel — the local copy that later outlives the shared row. Doing it + * through the door (rather than by reaching into the registry) is the point: + * this is exactly how a replica that only ever SERVED an entry ends up holding + * it. */ -async function seedServedDatasource( +async function seedServedDatasources( cluster: ReturnType, - name: string, + names: string[], ): Promise { - const res = await cluster.writer.protocol.saveMetaItem({ - type: 'datasource', name, item: datasourceBody(name), mode: 'publish', - }); - expect(res.success).toBe(true); + for (const name of names) { + const res = await cluster.writer.protocol.saveMetaItem({ + type: 'datasource', name, item: datasourceBody(name), mode: 'publish', + }); + expect(res.success).toBe(true); + } await settle(); - expect(await serves(cluster.writer, name)).toBe(true); - expect(await serves(cluster.peer, name)).toBe(true); + for (const name of names) { + expect(await serves(cluster.writer, name)).toBe(true); + expect(await serves(cluster.peer, name)).toBe(true); + } } describe('[#13609] the read door under measurement', () => { @@ -493,9 +576,9 @@ describe('[#13609] the read door under measurement', () => { await settle(); // `datasource` is `allowRuntimeCreate: true`, so `deleteMetaItem`'s - // `useRepoPath` fork takes the REPOSITORY exit — the one that emits. - // A code-only exit would publish nothing and this card's - // re-verification would be measuring a channel the delete never uses. + // `useRepoPath` fork takes the REPOSITORY exit — the one that emits. A + // code-only exit would publish nothing and this card's re-verification + // would be measuring a channel the delete never uses. expect(published).toHaveLength(1); expect(published[0].channel).toBe(METADATA_MUTATION_CLUSTER_CHANNEL); expect(published[0].payload.event).toEqual({ @@ -505,10 +588,10 @@ describe('[#13609] the read door under measurement', () => { it('the peer serves the row from its OWN registry, so the shared DB alone cannot answer for it', async () => { const cluster = makeCluster({ attach: false }); - await seedServedDatasource(cluster, 'billing_db'); + await seedServedDatasources(cluster, ['billing_db']); // The peer never wrote this row; it hydrated it by READING. This is the - // local copy whose lifetime the rest of the file measures. + // local, untimed copy whose lifetime the rest of the file measures. expect(cluster.peer.registry.listItems('datasource').map((i: any) => i.name)) .toEqual(['billing_db']); }); @@ -518,32 +601,93 @@ describe('[#13609] ⭐ the re-verification: how long does the peer keep serving beforeEach(() => { vi.useFakeTimers(); }); afterEach(() => { vi.useRealTimers(); }); - it('Arm B — WITH the landed publisher + bridge: BOUNDED at exactly one overlay-cache TTL window', async () => { + it('Arm B — WITH the landed publisher + bridge, the bridge DOES heal the peer registry', async () => { const cluster = makeCluster({ attach: true }); - await seedServedDatasource(cluster, 'billing_db'); + await seedServedDatasources(cluster, ['billing_db']); const res = await cluster.writer.protocol.deleteMetaItem({ type: 'datasource', name: 'billing_db' }); expect(res.success).toBe(true); await settle(); - // ⭐ The unbounded seam is SHUT: the peer's registry — the copy with no - // TTL — is healed within one bus hop, before the clock moves at all. - // This assertion is also what proves `settle()` is a sufficient drain. + // ⭐ The #13331 counter-evidence is vindicated, cross-node and on the + // DELETE direction: the publish crosses, the peer re-reads its own DB, + // finds no active row, and runs the same heal walk the writer ran. The + // peer's untimed registry copy is retired within one bus hop, before + // the clock moves at all. + // + // This assertion is also what proves `settle()` is a sufficient drain: + // an inadequate drain fails HERE, loudly, instead of silently inflating + // a prolongation in the cases below. expect(cluster.peer.registry.listItems('datasource')).toEqual([]); expect(cluster.peer.removedEntries).toContain('datasource|billing_db'); + }); + + it('⛔ …but the peer’s very next READ re-hydrates the deleted row from its own stale overlay cache', async () => { + const cluster = makeCluster({ attach: true }); + await seedServedDatasources(cluster, ['billing_db']); + + await cluster.writer.protocol.deleteMetaItem({ type: 'datasource', name: 'billing_db' }); + await settle(); + expect(cluster.peer.registry.listItems('datasource')).toEqual([]); + + // One read of the peer's own door, with the clock untouched. + expect(await serves(cluster.peer, 'billing_db')).toBe(true); - // …and what remains is the bounded residue, on the clock. + // ⭐ THE SEAM. The convergence retired the registry entry but did NOT + // retire the peer's overlay-cache entry — nothing on the receipt path + // touches the write epoch that keys it, and the peer does no writing of + // its own, so the pre-delete row set is still "fresh". `getMetaItems` + // then does what its cache's own header says it always does, hit or + // miss: it runs `hydrateOverlayIntoRegistry` over those rows. The + // deleted row is written straight back into the registry the bridge + // just cleaned — and the registry has no TTL. + expect(cluster.peer.registry.listItems('datasource').map((i: any) => i.name)) + .toEqual(['billing_db']); + }); + + it('⛔ Arm B measured, door READ during the residue window: UNBOUNDED, past 10 windows', async () => { + const cluster = makeCluster({ attach: true }); + await seedServedDatasources(cluster, ['billing_db']); + + await cluster.writer.protocol.deleteMetaItem({ type: 'datasource', name: 'billing_db' }); + await settle(); + + // `measureProlongationMs` reads the door once BEFORE advancing the + // clock — i.e. inside the residue window, which is what a replica under + // load does continuously. That read converts the bounded cache residue + // into an unbounded registry entry, so waiting the TTL out no longer + // helps: once the cache lapses the registry is the only source left, + // and it is the one now holding the deleted row. const prolongation = await measureProlongationMs(cluster.peer, 'billing_db'); - expect(prolongation).not.toBeNull(); - expect(prolongation).toBe(TTL_MS); + expect(prolongation).toBeNull(); + expect(await serves(cluster.peer, 'billing_db')).toBe(true); + }); - // Stated as the ruling asks for it: a NUMBER, and a bound. - expect(TTL_MS).toBe(30_000); + it('⭐ Arm B measured, door NOT read during the residue window: BOUNDED by one TTL window', async () => { + const cluster = makeCluster({ attach: true }); + await seedServedDatasources(cluster, ['billing_db']); + + await cluster.writer.protocol.deleteMetaItem({ type: 'datasource', name: 'billing_db' }); + await settle(); + + // Identical to the case above in every respect except one: nothing + // reads the peer's door while its stale cache entry is still fresh, so + // nothing re-hydrates the registry the bridge just healed. The entry + // lapses on its own and the peer converges. + // + // ⇒ The two cases together are the finding. The prolongation is not + // unconditionally unbounded and not unconditionally bounded: a SINGLE + // read of `/api/v1/meta/datasource` inside the residue window is what + // separates them, and that door under live traffic is read constantly. + vi.setSystemTime(Date.now() + TTL_MS); + await settle(); + expect(await serves(cluster.peer, 'billing_db')).toBe(false); + expect(cluster.peer.registry.listItems('datasource')).toEqual([]); }); it('Arm A — CONTROL, no bridge (the pre-fix shape): UNBOUNDED, still served past 10 windows', async () => { const cluster = makeCluster({ attach: false }); - await seedServedDatasource(cluster, 'billing_db'); + await seedServedDatasources(cluster, ['billing_db']); const res = await cluster.writer.protocol.deleteMetaItem({ type: 'datasource', name: 'billing_db' }); expect(res.success).toBe(true); @@ -556,32 +700,63 @@ describe('[#13609] ⭐ the re-verification: how long does the peer keep serving expect(cluster.peer.registry.listItems('datasource').map((i: any) => i.name)) .toEqual(['billing_db']); - // ⭐ This is the firing positive control. It proves the probe above CAN - // see a prolongation — Arm B's bound is the bridge's doing, not a - // harness that drops rows by itself. + // ⭐ The firing positive control. It proves the probe CAN see a + // prolongation and CAN see it end — the bounded case below returns a + // number on the same instrument — so the two unbounded readings above + // are measurements, not an instrument that never clears anything. const prolongation = await measureProlongationMs(cluster.peer, 'billing_db'); expect(prolongation).toBeNull(); + }); - // And it outlives the cache window by a factor of ten, which is why the - // original QA prolongation read LONGER than any TTL: the stale answer - // is not in anything that expires. - expect(await serves(cluster.peer, 'billing_db')).toBe(true); + it('⭐ SCOPED kernel: the same delete IS bounded — at exactly one overlay-cache TTL window', async () => { + const cluster = makeCluster({ attach: true, environmentId: 'env_prod' }); + await seedServedDatasources(cluster, ['billing_db']); + + // On a scoped kernel BOTH hydration seams are gated off — the read-side + // loop and the write-through alike — so no replica ever holds a local + // registry copy of an overlay row, and the only local source is the + // overlay cache, which does expire. + expect(cluster.peer.registry.listItems('datasource')).toEqual([]); + + const res = await cluster.writer.protocol.deleteMetaItem({ type: 'datasource', name: 'billing_db' }); + expect(res.success).toBe(true); + await settle(); + + // ⭐ The bound, stated as a number: one window, and the constant is + // imported rather than retyped so this tracks whatever ships. + const prolongation = await measureProlongationMs(cluster.peer, 'billing_db'); + expect(prolongation).toBe(TTL_MS); + expect(TTL_MS).toBe(30_000); + + // This is the residue the ruling permits ("listCache TTL is the one + // bounded residue that may legitimately remain") — refined to the cache + // that actually holds it at this door. + expect(await serves(cluster.peer, 'billing_db')).toBe(false); }); - it('negative control — an unrelated datasource is untouched by either arm', async () => { - for (const attach of [true, false]) { - const cluster = makeCluster({ attach }); - await seedServedDatasource(cluster, 'billing_db'); - await seedServedDatasource(cluster, 'analytics_db'); + it('negative control — an unrelated datasource is untouched, in every arm', async () => { + for (const shape of [ + { attach: true, environmentId: undefined }, + { attach: false, environmentId: undefined }, + { attach: true, environmentId: 'env_prod' }, + ]) { + const cluster = makeCluster(shape); + await seedServedDatasources(cluster, ['billing_db', 'analytics_db']); await cluster.writer.protocol.deleteMetaItem({ type: 'datasource', name: 'billing_db' }); await settle(); vi.setSystemTime(Date.now() + TTL_MS * (WINDOWS_PROBED + 1)); await settle(); - // The delete reached exactly one name on the peer, whichever arm. + // The delete reached exactly one name on the peer, in every shape — + // so the prolongations measured above are about the deleted row and + // not about a peer that has stopped answering. expect(await serves(cluster.peer, 'analytics_db')).toBe(true); - expect(await serves(cluster.peer, 'billing_db')).toBe(attach ? false : true); + // No read landed inside the residue window here (the clock jumps + // straight past it), so the bridged shapes converge and only the + // unbridged one prolongs — the same discriminator as above, seen + // from the other side. + expect(await serves(cluster.peer, 'billing_db')).toBe(!shape.attach); } }); }); From 5f404b94afcc7cd5e2d014a7a0cb630765bc0a90 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 2 Sep 2026 06:21:53 +0000 Subject: [PATCH 3/3] chore: record the new measurement file's engine doubles in the pinned ledger Gate scaffolding for check:engine-double-contract, produced by `node scripts/check-engine-double-contract.mjs --write`. The three rows are the delete/findOne/update doubles of the #13609 measurement harness, all already routed through assertEngine*Dispatch; the ledger just has to learn about the file or it never protects it. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_0112hMx9hjJ9BgB28X97DS68 --- scripts/engine-double-contract.pinned.json | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/scripts/engine-double-contract.pinned.json b/scripts/engine-double-contract.pinned.json index bb477515dd..b28325d7dd 100644 --- a/scripts/engine-double-contract.pinned.json +++ b/scripts/engine-double-contract.pinned.json @@ -416,6 +416,21 @@ "verb": "update", "pinned": 1 }, + { + "file": "packages/metadata-protocol/src/protocol.datasource-delete-prolongation.test.ts", + "verb": "delete", + "pinned": 1 + }, + { + "file": "packages/metadata-protocol/src/protocol.datasource-delete-prolongation.test.ts", + "verb": "findOne", + "pinned": 1 + }, + { + "file": "packages/metadata-protocol/src/protocol.datasource-delete-prolongation.test.ts", + "verb": "update", + "pinned": 1 + }, { "file": "packages/metadata-protocol/src/protocol.delete-object-registry-unregister.test.ts", "verb": "delete",