doc: design decoupled coordination

[aljoscha]

Status

M2 integration remains active. Native ownership stays enabled.
Head e8ae0964 is rebased onto upstream f5217bd2d7, with a linear,
consolidated history. CI137885, at the same commit,
is verifying the reviewed upstream replay, approved frontend OCC correction and
scoped history-observation repair. CI137869 found four explicit-Arc-clone lint
errors in the new tests, now corrected, and upstream merge conflicts, now resolved.
No OCC runtime success is claimed from that superseded run. Independent reviews
found no blocking correctness or integration defect. Scoped Rust formatting,
Ruff, Black and diff checks passed locally. CI137872 passed Clippy, formatting,
doc checks and early Console checks before cancellation. Its runtime jobs never
started. Upstream subsequently moved to d9a2c5dc40, including removal of legacy
coordinator RTW. New merge conflicts are under separate assessment while runtime
verification continues, without another immediate rebase.

CI137825 verified both
restart shards, the blind-INSERT catalog read-protection fixture and Cluster1's
metrics correction. It still failed Cargo (request/outer timeouts), parallel
checks (OCC conflicts plus DDL timeouts), SLT1/2 (job timeouts), hydration history,
Console and the deferred pre-feature conversion. These remain distinct findings.
CI137778 passed Cargo,
all five SLT shards, Short Zippy, native outage, compatible-version warm promotion,
bounded publication and both imported hydration-stability workflows. Earlier
passes remain evidence, not explanations of subsequent failures. Heavy checks
run in CI only.

The approved corrections are implemented:

• Both retention fixtures use their scoped 5s heartbeat / 25s grace from startup,
  wait for the retired incarnations, then retain the unchanged advancement checks.
• Managed adoption validates existing replicas before DDL commits scheduled RF0.
  Equivalent ALTER/RESET paths agree without scheduler writes to shared intent.
• Promotion includes a bounded, reported interval through SQL readiness. The
  first query keeps its three-second limit, with no warm-up query.
• The obsolete native peek-history regression is removed, not replaced with
  zero-counter assertions. Meaningful query cleanup coverage remains.
• Upstream OIDs are preserved. The two unreleased raw sources use fresh OIDs.

Upstream integration preserves native ownership in both standalone and unified
compute/storage hosting. Storage retains connection state across guest turns,
while the host owns scheduling and the common command lane. Independent source
review found no blocking issue. Rust formatting, Cargo manifest policy, Python
syntax, Ruff and Black checks passed locally. These are not runtime proof.

Upstream's replica-reported hydration stability check uses the canonical native
background client and real read protection. Fresh scheduling timestamps are
obtained off the coordinator loop. Native catalog inventory, local progress and
existing exemptions remain authoritative. The imported adapter-restart fixture
uses actual environmentd-only process supervision rather than assuming native
prewarming restarts on DDL. Independent source review found no blocking issue.
Both imported caught-up stability workflows passed in CI137762: adapter restart
and replicas without dataflows. These were executed independently of warm handover.

The latest replay preserves all 136 local commits from the preceding linear
branch. Range-diff changes are confined to three conflict resolutions: logging
imports, the certified-execution tracing attribute, and deterministic cancellation
fixtures. The imported legacy drop-triggered restart workflow is retired, not
used to reintroduce restart-based native ownership. Preflight ID-set boundary
coverage remains. The pre-replay tree is preserved in a local archival tag.

The old merge's substantive resolutions and the side branch were audited.
Consolidation preserves the exact reviewed tree, with no merge commits in the M2
range. Designer decisions remain separate. There is one active M2 branch, with
local archival tags preserving the original history and deferred side-branch
work. The useful alternating-version append/readback coverage was retained at
the Persist authorization boundary. The chosen upstream already includes the
MinIO source-build fix. The approved Docker Hub workaround remains separate.

Design ·
Current handoff

Implementation boundaries

• Shared catalog loading and implications reconstruct native maintained execution
  without a clusterd dependency on the adapter. Query execution has independent
  ownership and real read protection.
• Index creation commits selected plans, justified bounds and logical/actual-import
  protection together. SELECT and EXPLAIN share protection acquisition before
  installation. Catalog writers use the shared oracle, serving requests validate
  definition/configuration context, and statement-specific timeout scope remains.
• Deliberate replica carryover preserves public ReplicaIds. Membership, settings,
  plan ownership and runtime state are deployment-qualified. Read-only participation
  does not grant output-write authority. Externally authorized promotion retains
  warmed native execution.
• Typed durable admission governs Persist format compatibility, including fresh
  read-like initialization and explicit upgrades. Actual binary identity stays
  separate. Participants follow committed authorization live. Compatible Persist
  writers may overlap through existing protocols, without a generic append fence.
• Current observations follow active catalog authority. History survives promotion,
  with processes summed within deployment before existing chart aggregation.

Verification and remaining work

Verified corrections:

• Catalog-only updates stage table keepalives without waiting on the coordinator.
  The isolated SQL regression passes with the committer parked. Nonempty builtin
  visibility, oracle completion and protection remain unchanged.
• Real comment-ID collision coverage and the exact EXPLAIN TIMESTAMP ready output
  pass. The latter retries boundedly for asynchronous table progress.
• Earlier physical-since observation preserves the restart admission bracket.
  The complete catalog read-protection restart passed in CI137762.
• CI137778 passes all SLT shards, including unchanged scheduling membership/audit
  assertions and the corrected webhook ID. RBAC streaming diagnostics are removed.
• Short Zippy passes CI137778. Its preceding run completed seven promotions but
  exhausted the budget restoring the catalog shard. That history-amplified restore
  cost and readiness growth from 42s to 363s remain observations, not a proven plateau.

Verified in CI137825:

• DROP rewrite preparation refreshes the owner's committed requirement when
  acquisition observes a later readable floor. Structural revision changes still
  reject stale work. It never advances a requirement to accommodate an import,
  and missing committed requirements fail closed. Independent review found no
  safety gap. CI137817 passed existing prepared-DDL and selected-plan SQL tests,
  without a new catalog-writer harness or unused failpoint.
• Compaction input generation uses blind INSERTs instead of read-dependent UPDATEs.
  It still produces real batches without adding input read protection. The physical
  cutoff, grace, deadlines and exact-output assertions are unchanged.
• Cluster1 exposed a transient command-history assumption: reduction can fold all
  compaction commands into dataflow frontiers. Remove the positive retained-command
  count assertion, preserving dataflow and runtime measurements. Ruff and Black pass.

Under CI137885 verification:

• OCC chooses subscribe-certified targets capped by oracle progress. Persist's
  txns CAS arbitrates availability, without oracle-only stale-target rejection.
  Conflicts return the actual txns upper and the frontend refolds diffs before
  retrying. Freshness, future-time policy, linearization, fencing, target validation
  and cancellation remain intact. Catalog completion before acknowledgement is
  explicitly load-bearing in the corrected design and adapter guide.
• New committer tests cover targets at/below oracle progress and actual-upper
  conflict replies without silent retries or completion. Existing SQL, concurrency,
  future-refresh and query-cleanup regressions remain unchanged. No new harness or
  precommit observation hook was added. Runtime results are pending.

Remaining integration failures:

• Both parallel shards timed out in CI137778. Their scoped RTW phase diagnostics
  show repeated global-oracle timestamp rejection despite advancing subscribe
  progress, plus separate pre-execution catalog replanning. Baseline has the same
  OCC rule, but this branch adds independent catalog oracle writers. A bounded
  timestamp-selection correction is approved and in CI137885. Queue/progress delays
  under contemporaneous swapping remain separate, without an identified wait owner.
  CI137762 parallel2 also hit shared-cgroup memory pressure and a late clusterd OOM,
  after all 22 terminal SQL timeouts. The OOM alone does not explain their origin.
• The DROP preparation race is verified above. In CI137778, catalog read-protection
  restart passed its admission bracket, then its helper
  UPDATE timed out during reclamation. The new blind-write pump isolates that
  fixture, without treating independent RTW stalls as fixed.
• Console E2E/prod repeats a five-second new-password dialog timeout in CI137778.
  Its earlier trace
  showed an incomplete Frontegg request and no SQL requests. No assertion changed.
• Pre-feature upgrade conversion remains deferred and its safety guard unchanged.

Temporary RTW debug events are enabled only for parallel CI. Scheduling SLT also
streams its real-time window. Remove diagnostics once their questions are answered.

Keep two observations visible: CI137825 hydration-history capture shows a zero-row
collector visit followed 24s later by a one-row append, beyond the approximately
20s query budget. Delayed sampling is supported, but the raw completion timestamp
and appended object identity are absent. Only that initial history observation gets
a scoped 60s retry budget, then restores the default. Temporary collection logs
are removed. No history semantics changed. Catalog-cluster
hydration progress during repeated handover also remains an observation.
CI137676 passed Cargo,
Clippy, Testdrive4 and Short Zippy, but was not an all-green build. A passing history
run with diagnostics does not explain the earlier failure.
CI137666 passed all 4,147
Cargo tests, both Clippy jobs, Cluster1/2/5, Restart1/2 and cluster isolation.
Its Zippy run completed five promotions without ingress panics, then exhausted
its total budget during generation6 catalog hydration. Those checkpoints are
historical evidence, not substitutes for current CI.

Completed evidence

• Native outage/recovery: maintained source, MV and Kafka-sink output and
  physical compaction continue without the adapter. Replicas reconstruct during
  the outage, adapter restart resumes exact queries, and zero-replica retention
  advances before a historical query executes through the reconstructed index.
  Native recovery proof.
• DDL/protection: nonempty prepared rewrites survive metadata-only contention.
  Structural invalidation and reclaimed authority prevent stale commits. Separate
  writer/replica executables exercise the development plan namespace and version
  rejection. CI135583.
• Compatible-version warm handover: CI137723 native promotion
  passed with genuinely compiled v26.46.0-dev.0 and v26.47.0-dev.0 binaries from
  86d5cdb9. Promotion through SQL readiness took 8.736s, followed by an actual
  first query in 0.451s, without a warm-up. Indexed and table reads matched
  20 million rows, then 40 million after resumed ingestion. This is the existing
  Kafka UPSERT handover fixture, not the entire 0dt suite. Historical passes include
  CI137654 and
  CI137662.
• Iceberg: overlap-induced failure, native reconstruction and six committed
  records without duplicates. CI135350.
• Bounded throughput: two replicas, one-second publication and two offered
  DDL pairs/second across three 30-second windows, with unchanged shared-view topology.
  CI135429.

Objects	Completed/offered pairs	Pair p50/p95	DROP p50/p95
100	180/180	101.77/199.64 ms	34.18/66.81 ms
1,000	180/180	131.35/259.67 ms	42.62/138.71 ms

Shared consensus state-diff traffic was 3.33/3.40 MB. Shared retained batch bytes
grew 6.21→26.10 MB and 44.58→63.37 MB. The initial zero-frontier reader advanced.
Ordinary lease-refresh lag is an accepted cost. A steady-state plateau was not
demonstrated. These completed proof scopes stay closed.

Scope and limitations

Managed-cluster replica-targeted MV warm handover is excluded and rejected before
promotion. Public mz_cluster_replicas remains shared, materializable and filtered
to active deployment. Internal routing and readiness use local membership.
Table/WAL work, webhook ticking, provisioning, adapter-owned introspection writes
and shard finalization remain adapter responsibilities.

Kafka's roughly five-minute abandoned-incarnation takeover remains an accepted
provisional limitation, not a production failover target. Production grace and
Persist leases are unchanged. MV replacement application remains asynchronous.
Development plan namespaces remain writer-specific, and provisioners must pair
compatible siblings. Semantic-version equality and decoding are not compatibility
proofs. Catalog fencing, read protection and external-sink safeguards remain required.

Pre-feature environment conversion, older branch-built catalogs and arbitrary
concurrently serving adapters are outside M2. The pre-feature Source-to-View
upgrade guard is not weakened or silently skipped. CI137723 hits that guard
upgrading from v26.45.0-rc.4, separate from the native participating-version proof. Independent query-client
isolation remains M3.

[aljoscha]

Commit 9d5e1a9 connects storage-backed catalog read protection to committed delivery. Bounds and maintained requirements have separate durable records, final-state transaction validation rejects incompatibility and regression, and the adapter returns admission errors before its fatal commit boundary. Committed bounds reach storage through implications. Dry runs model collection births and drops without controller effects.

This is an intermediate milestone 1 change, not activation. SQL creation still needs to produce initial bounds and logical-input requirements with secured readability, including shared-shard initialization. Recovery and durable-progress publication remain unwired.

Validation: formatting, targeted compilation, and the durable admission/reopen and adapter admission/dry-run tests pass. Independent review findings are resolved. Regular PR CI is still pending: https://buildkite.com/materialize/test/builds/134033