feat: multi-authority space host with dynamic discovery and projection syncer - #233
Open
rishibalakrishnan wants to merge 56 commits into
Open
feat: multi-authority space host with dynamic discovery and projection syncer#233rishibalakrishnan wants to merge 56 commits into
rishibalakrishnan wants to merge 56 commits into
Conversation
Adds an IndexMutation journal written alongside every synced batch, independent per-projector cursors with dead-lettering, the Projector trait, and typed routing from index mutations to sync events.
DAEMON_FEEDS_URL / DAEMON_APPVIEW_URL (with their service DIDs) give every discovered space a worker that drains each destination on its own cursor; an empty URL leaves the daemon index-only.
Implements RepoStore over per-account store.sqlite files carrying the space_* tables, transcribing the pinned oracle's apply-writes semantics: server-minted revisions, per-write swap checks, record-exists and record-not-found refusals, oplog append and compaction. The harness now drives both sides through per-batch scripts, compares refusal reasons as well as contents, and adds S6.
Adds a second comparator over the space_* rows of both store files, with server-minted revisions, oplog ids and creation timestamps replaced by first-appearance placeholders, plus assertions that every revision is a valid TID and that oplog order agrees with revision order. Paged reads are now walked page by page on both sides. S7 (unicode and prefix-colliding keys), S8 (large record), S10 (two spaces, one author) and S11 (pagination) join the scoreboard: 10/10.
S9 drives both stores past a three-row oplog window, so the shim now has to match the oracle's compaction, its oplog floor, and its refusal to serve history that fell out of the window. Every scenario additionally reopens the shim's own file with the oracle's SpaceStore and compares what it reads (S13), and probes history since every revision each side minted. S12 records the one accepted divergence: the oracle indexes blob references, the shim's storage keeps the bytes without indexing them and its write path refuses blob-bearing records outright.
Converts a deployed multi-tenant store into per-account actor stores, carrying the LtHash state and every oplog row id across verbatim so readers see the same digest and syncers resume on the cursors they already hold. Record revisions, which the source schema does not carry, come from the newest operation that left the record at its current CID. S14 builds a fixture in the frozen source schema, converts it, and reads the result back with the oracle: heads, records, oplog ids and resume-from-cursor all verified. Scoreboard: 13/13 (+1 documented divergence).
The in-memory store no longer no-ops a delete of an absent record, and an update of an absent record is refused rather than upserted, matching what the actor-store backing and the pinned oracle do. WriteOutcome::Noop goes with it, so deleteRecord always records the write.
The two-process gate found three response-level divergences from rsky-pds that the in-process suite could not see: - RecordExists, RecordNotFound, InvalidSwap and HistoryUnavailable now answer with their own names and the oracle's status instead of being collapsed into InvalidRequest or answered with 409/410. - listRepoOps hydrates an op's value only when the op's cid still names the stored record, so a superseded op no longer carries the current record's value. - The local write path records the commit digest in the writer set, so listRepos returns the hash the oracle returns.
One command runs both real binaries and prints per-endpoint parity: run.sh builds rsky-pds from a detached build-only worktree at the pinned oracle revision, builds the space host from the working tree, then fires the same record script at each over XRPC and points the existing Layer 1 stored-row comparator at the two store files. The gate supplies the only support service itself: a stub DID directory on a loopback port, and the access tokens and space credentials each server demands. No database, container or network is involved. The four methods only rsky-pds routes are probed against it alone and reported as a surface difference rather than a parity failure; the probe is deliberately invalid so it cannot write to one side only.
18/18 checks equal (+1 documented divergence) with both real binaries running. Records the surface difference a client gains when it moves from the space host to rsky-pds, the three divergences the gate found, and the falsification evidence for each of its three comparison layers.
The converged space host deploys over data the legacy host wrote. This runs that upgrade with real processes — legacy host, real daemon, converter, converged host, same daemon — and asserts the daemon resumes on its existing cursors: exactly the operations after the cursor, once each, with no cursor refusal, divergence or full-state recovery. A cold daemon on the converted store is compared against it.
The managing app resolves an inbound service JWT against the issuer's #atproto verification method, so a call signed with the space key is rejected. Mint checkUserAccess and ackHostRegistered through PdsServiceJwtIssuer instead, and refuse at boot when a pinned authority under the managing-app policy has no actor-store key.
Service auth and delegation tokens are keyed on #atproto, but the shared resolver preferred #atproto_space and so selected the wrong key for any DID publishing both. Credential verification keeps the space-first order.
SPACEHOST_ACTOR_STORE_DIR was both the signing-key source and the ActorStoreRepos write target, which made the shim write space tables into the PDS's own actor files — the rejected Option B, and impossible where that directory is mounted read-only. Writes now go to SPACEHOST_SPACE_STORE_DIR, the same path is refused at boot, and a store that already carries the base schema is adopted rather than re-migrated. Both gates only ever ran the two directories as one path and never opened a store the PDS created, so neither could see this: Layer 1 gains those two cases and Layer 2 now runs them separated and asserts the key directory is never written to.
WebCrypto emits a high-S ECDSA signature about half the time, so verifying a DPoP proof or access token under the repo-commit signature policy failed roughly every other request from a legitimate client. Verify those through shim-local helpers that pass allow_malleable_sig, and keep the strict verifiers for repo commits, where a malleable signature would be a second valid signature over the same content.
SpaceError::BadSignature reads "commit signature verification failed", which is accurate for a repo commit and misleading on a DPoP proof — where it kept surfacing and kept sending readers to look at commits.
allow_malleable_sig only waives the encoding check on this curve, and libsecp256k1 refuses a high-S signature regardless, so roughly every other DPoP proof from a client whose key is secp256k1 was rejected. The DER path is left as it was. Tests now pin the asymmetry on both curves: a client's high-S proof verifies, and the commit path still refuses it.
A projectRecords 401 means the author is not admitted to the space, which is state, not a bad batch: it flips when a membership write propagates. Three fast attempts then dead-lettered the batch permanently, so a moment of FGA lag left a record unprojected for good. Denials now get their own error class, their own durable counter beside the poison one, and a slow lane: re-attempted once per sweep, parked after DAEMON_DENIAL_PARK_AFTER sweeps so a legitimate refusal still stops. A denied author's later batches wait behind the denied one, or the cursor would advance past it. Poison keeps its fast dead-letter unchanged.
The converged space host has required SPACEHOST_SPACE_STORE_DIR since the two directories were separated; the gate still passed one path and so failed at boot. The legacy era predates the split and keeps one directory.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ports the standalone space host and syncer daemon onto the merged spaces interop surface. The host serves permissioned space repos for multiple authorities in one process, resolving each authority's signing key from the PDS actor store, with DPoP-bound credentials (ES256/ES256K) minted for an allowlisted set of service identities. The daemon discovers spaces dynamically from the managing app, syncs and verifies their oplogs, and projects records to the feed service and appview ingress with journaled, per-projector cursors.
Test plan
cargo test -p rsky-space-host(121 tests)cargo test -p rsky-daemon --lib(92 tests)cargo test -p rsky-pds --lib