feat(metadata-protocol,service-cluster): fan runtime metadata mutations out to peer replicas

.changeset/metadata-mutation-cluster-fanout.md

@@ -0,0 +1,52 @@
+---
+"@objectstack/metadata-protocol": minor
+"@objectstack/service-cluster": minor
+---
+
+feat(metadata-protocol,service-cluster): fan runtime metadata mutations out to peer replicas — a runtime-authored object no longer answers OBJECT_NOT_FOUND on every replica that did not perform the write (#13331)
+
+Measured on a live 3-replica EE deployment (ADR-0018 compose, redis driver):
+an object authored through `PUT /api/v1/meta/object/...` persisted to the
+shared `sys_metadata` (so `/api/v1/meta/*` answered 200 fleet-wide) but
+registered with the ObjectQL engine registry of the writing replica only —
+`/api/v1/data/<object>` answered a hard 404 `OBJECT_NOT_FOUND` on the other
+replicas, indefinitely (200 concurrent creates through the LB: 67×201 /
+133×404; a boot-loaded control object: 0 errors; the only recovery was a full
+fleet restart). The runtime authoring path lives entirely in the metadata
+protocol and never touches the metadata service, so the existing
+`metadata.changed` bridge — even when attached — never heard these writes.
+
+Maintainer-ruled design (2026-09-01, Option A):
+
+- **Publisher at the producer choke point.** The protocol's post-persistence
+  mutation funnel (`saveMetaItem` / `publishMetaItem` / `deleteMetaItem` —
+  the same seam `onMetadataMutation` subscribes) now also publishes the
+  mutation's ADDRESS on a new cluster channel `metadata.mutated`
+  (`METADATA_MUTATION_CLUSTER_CHANNEL`, payload
+  `ClusterMetadataMutationPayload`). Drafts are not published — they never
+  enter any replica's registry.
+- **Peers converge from their own DB read.** On receipt, a replica re-reads
+  the row from its OWN `sys_metadata` and re-runs the registry write-through
+  (active row present) or the delete heal walk (no active row). The payload
+  is a signal, never trusted content — the shared database stays the single
+  source of truth, and duplicate or out-of-order delivery converges to the
+  row's current state by construction. After convergence the event replays
+  into the replica's local `onMetadataMutation` listeners (never
+  re-published), so boot-cached consumers such as the authored hook/action
+  re-bind re-sync on peers exactly as they do on the writer.
+- **New attach seam, mirrored from the shipped bridges.**
+  `ObjectStackProtocolImplementation.attachMetadataMutationPubSub(pubsub,
+  nodeId)` — idempotent on the `(pubsub, nodeId)` pair, with loopback
+  suppression via `originNode`, shaped after
+  `MetadataManager.attachClusterPubSub()` and the engine's
+  `attachAuthzInvalidationPubSub()`. `MetadataClusterBridgePlugin` late-binds
+  it at `kernel:ready` as a second, independent lane beside the existing
+  metadata-service lane — the boot shape that lacks a manager-backed
+  `metadata` service (the TS-config host-config boot, exactly the shipped EE
+  shape) is the one that needs this lane most. The new lane skips the
+  in-process memory driver (nothing to fan out to), the guard the authz
+  sibling already carries.
+
+No shipped driver exceeds at-most-once delivery, so a lost message still
+degrades to the pre-existing staleness bound (heal at next boot); this
+channel narrows the window from "until restart" to one network hop.

content/docs/permissions/system-context.mdx

@@ -112,7 +112,7 @@ that silently does not happen.
 | 18 | **`readonly` strip bypassed — UPDATE, single row** | objectql | Get: a `readonly` field CAN be written. Lose: the protection that stops a caller seeding e.g. `approval_status` | `objectql/src/engine.ts:10914` |
 | 19 | **`readonly` strip bypassed — UPDATE, bulk/predicate** | objectql | Same, on the multi-row path | `objectql/src/engine.ts:11076` |
 | 20 | **`readonly` strip bypassed — INSERT (engine pass)** | objectql | Same, on create | `objectql/src/engine.ts:9772` |
-| 21 | **`readonly` strip bypassed — INSERT (protocol ingress)** | metadata-protocol | `isSystem` is the **only** exemption here. `preserveAudit` is deliberately not read on this path (#6640) — a non-system historical import is still stripped on create | `metadata-protocol/src/protocol.ts:1737` |
+| 21 | **`readonly` strip bypassed — INSERT (protocol ingress)** | metadata-protocol | `isSystem` is the **only** exemption here. `preserveAudit` is deliberately not read on this path (#6640) — a non-system historical import is still stripped on create | `metadata-protocol/src/protocol.ts:1741` |
 | 22 | Strict-drop refusal never fires | objectql | Lose: a caller that opted into loud refusal gets **silence** — strict refuses exactly what the strip would have taken, and the strip took nothing | `objectql/src/engine.ts:9809`, `readonly-strict-errors.ts:66` |
 | 23 | **Referential-integrity check skipped** | objectql | Get: writes proceed against unreachable/unresolvable targets. Lose: an `isSystem` caller can write a **dangling reference** | `objectql/src/engine.ts:5730` |
 | 24 | Tenant-audit warning silenced; `bypassTenantAudit` threaded to the driver | objectql | Get: unscoped system writes stop warning. Lose: the signal that would flag a genuine user-path scoping bug | `objectql/src/engine.ts:3599`, `:3609`, `:3636` |

packages/metadata-protocol/src/index.ts

@@ -135,6 +135,13 @@ export type { UninstallCleanup, UninstallCleanupOutcome } from './protocol.js';
 // against the producer's contract instead of restating it locally.
 export type { DeletePackageRequest, DeletePackageResponse } from './protocol.js';
 export type { MetadataMutationEvent, MetadataMutationProjector, MutationProjectionOutcome } from './protocol.js';
+// [#13331] The cross-node half of the mutation notification: the cluster
+// channel the protocol publishes post-persistence mutations on, and its
+// address-only payload. Exported for the bridge's tests and for any peer-side
+// consumer that must speak the channel by name — the payload is a SIGNAL,
+// never trusted content (receipt re-reads `sys_metadata` locally).
+export { METADATA_MUTATION_CLUSTER_CHANNEL } from './protocol.js';
+export type { ClusterMetadataMutationPayload } from './protocol.js';
 // [#10219] The per-item publish notification the host bridges to the
 // kernel-wide `metadata:reloaded` announce. Exported for the same reason its
 // mutation sibling is: the subscriber lives in another package.