feat(messages): notify substrate, addressing extraction and history backfill (CHOO-1436) by amaudruz · Pull Request #366 · sandbox-quantum/switch

amaudruz · 2026-09-03T21:57:20Z

Step 5 of the Tuwunel-replacement plan, plus outstanding items 1 and 3. Stacked on #365 (worktree-message-store).

Four commits, each independent enough to read on its own.

1. The notify substrate — `857807af`

A trigger on messages fires pg_notify for every insert. A MessageListener holds one LISTEN for the process's life and fans announcements out in-process to whoever subscribed to a room.

The announcement is a hint, never the payload. It carries a room, a position and a row id; the subscriber reads the rows it has not seen. Under that rule everything awkward about Postgres's notification queue — not durable, not replayable, delivered at most once, only to connections that happened to be listening — stops being a correctness problem, because the worst a lost announcement can do is delay a read the next one triggers anyway.

Three consequences that are load-bearing rather than incidental:

Announcements coalesce per room. A burst during one handler run is one wake-up, and the handler still sees every row, because it works from its own cursor. That is also why there is no backpressure to apply and no queue to overflow.
Reconnecting wakes every subscriber. Announcements missed while disconnected are gone for good, so the listener treats a reconnect as "everything may have moved" rather than making each consumer detect its own gap.
It is a trigger, not a call in the writer. A future writer cannot forget it, and it commits with the row or not at all.

delivery_cursors lands with it — the table step 3 deferred. GREATEST on advance, never an assignment: two deliveries can finish out of order, and rewinding a cursor redelivers.

Deviation from the plan: one channel, not one per room. A per-room channel still needs its LISTEN issued before the first row it should catch, and closing that race means reading the table on subscribe — which the consumer does anyway. The correctness machinery is identical either way; the churn buys only fewer wakeups. Reversible: the fan-out is the only thing that would change.

2. Addressing decided from stores, not from a client — `3e0f0162`

Pure refactor. "Is this message for me?" lived on AgentClient and reached every answer through a live client. A dispatcher reading rows has none of that, and the tempting shortcut is a second implementation for the second source.

That is the one thing not to do — addressing is how the scoped addressing policy is enforced, and a policy enforced by one path and not the other is worse than no policy. So the rules move to switch_core/delivery/addressing.py and take a message as data. AgentClient builds one from a bus event; the dispatcher will build one from a row.

Tests follow the logic rather than being adapted to it.

3. The permission events, deleted — `9cbf173e` (outstanding #3)

Scoping the log flagged com.switch.permission.request / .response as never-sent. That left an open question: dead code, or a producer outside switch_core — which would have holed the outbound-only capture premise and meant the denylist was dropping real traffic.

Dead, since the initial import. No sender in core, the connectors, the console or the gateway; no generic "post an arbitrary event type" path; none in the history. On receipt they dispatched to handlers no subclass overrides, so an arriving one was dropped without reaching the buffer.

4. History backfill — `cd4b38a2` (outstanding #1)

Step 4 turned pre-recording history from unqueried into invisible: 731 of 739 bridged messages on the first deployment. just backfill-messages walks each room to its start and writes what has no row.

Reconstructed rows are numbered below zero. A message from last month cannot take a number above one from this morning, and cannot take one among the live rows either — renumbering would move every cursor pointing at them. Counting down from zero keeps the order right and makes the sign say something: positive was recorded as it happened, negative was reconstructed afterwards.

The sign is load-bearing. A delivery cursor starts at 0, so backfilling a year delivers none of it — the only sane answer. The notify trigger skips these rows for the same reason.

Idempotent by design: each page commits on its own, transport_event_id is unique, so a walk that fails halfway is safe to just re-run. --dry-run walks exactly as far and writes nothing.

sent_at is the bus event's timestamp, not the walk's clock — otherwise a year of history piles onto tonight and every windowed read is wrong about all of it.

What is NOT here, and why

The dispatcher. Reading rows and routing them to agents instead of AgentClient.on_message doing it off the Matrix sync loop.

It has a dependency the plan does not account for, and I would rather name it than force it at 3am with no live stack to check against:

The log is scoped to the conversation — m.room.message, com.switch.command, arrivals. Commands, task events, mediation RPC, telemetry and runtime state are deliberately not in it, so a dispatcher driven by messages cannot carry them. AgentClient therefore keeps its Matrix sync loop regardless, and the puppet model cannot be deleted until those types have a transport of their own. Step 6 removes the twelve RPC types; the rest need a decision.
The no-live-session auto-reply currently happens inside on_message and sends a message, so moving it to a dispatcher means giving the dispatcher a sending identity.
Multi-attachment grouping is stateful across events in AgentClient.

So the cutover wants either step 6 first, or a second notify path for the events the log does not keep. That is a plan question, not a coding one.

Testing

2354 pass, ruff and mypy clean. The trigger and the LISTEN are tested against real PostgreSQL — a fake would prove the fan-out works and say nothing about whether the database ever speaks. Backfill numbering, idempotence and scoping likewise run against real Postgres.

Not yet run against the real Matrix stack (the integration suite), in common with #363 and #365.

🤖 Generated with Claude Code

…1436) The first half of moving delivery off Matrix. Nothing consumes this yet; it is the substrate the dispatcher will sit on. A trigger on `messages` fires `pg_notify` for every row, and a listener holds one `LISTEN` for the process's life and fans announcements out in-process to whoever asked about a room. **The announcement is a hint, never the payload.** It carries a room, a position and a row id; a subscriber reads the rows it has not seen. Under that rule everything awkward about Postgres's notification queue — not durable, not replayable, delivered at most once, and only to whoever happened to be connected — stops being a correctness problem, because the worst a lost announcement can do is delay a read the next one will trigger anyway. Putting the message in the payload would make the queue authoritative, and it is not built to be. Three consequences that are load-bearing rather than incidental: - Announcements coalesce per room. A burst during one handler run is one wake-up, and the handler still sees every row, because it works from its own cursor. That is also why there is no backpressure to apply and no queue to overflow. - Reconnecting wakes every subscriber. Announcements missed while disconnected are gone for good, so the listener treats a reconnect as "everything may have moved" rather than making each consumer detect its own gap. - It is a trigger rather than a call in the writer, so it cannot be forgotten by a future writer, and it commits with the row or not at all. A listener never hears about a row it could not then read. `delivery_cursors` lands with it — the table step 3 deferred. It persists what the event buffer held in memory, so an agent's position outlives the process serving it. `GREATEST` on advance, never an assignment: two deliveries can finish out of order, and rewinding a cursor redelivers. **Deviation from the plan: one channel, not one per room.** A per-room channel still needs its `LISTEN` issued before the first row it should catch, which is a race the consumer closes by reading the table — so the correctness machinery is identical either way and the subscription churn buys only fewer wakeups. The in-process fan-out does the routing instead. The listening connection is built outside the application pool. It is held, not borrowed; pool recycling would drop the subscription, and the silence afterwards is indistinguishable from a quiet room. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…(CHOO-1436) Pure refactor, no behaviour change. It is what has to happen before delivery can be driven from the message table rather than from a Matrix sync loop. "Is this message for me?" lived on `AgentClient`, and every answer went through a live client — the agent came off `self`, the mxid off the transport session, the stores off the client's own wiring. A dispatcher reading rows out of `messages` has none of that, and the tempting shortcut is a second implementation for the second source. That is the one thing not to do. Two code paths answering "is this for me?" would be a security bug waiting for a rewrite to expose it: addressing is what the scoped addressing policy is enforced through, and a policy enforced by one path and not the other is worse than no policy. So the rules move to `switch_core/delivery/addressing.py` and take a message as data — a sender, a body, a content dict. `AgentClient` builds one of those from a bus event; the dispatcher will build one from a row; both get the same answer because it is the same code. The two questions stay apart, as they were: **addressed** is intent (name, room alias, a role held live, or the other party in a direct chat), and **permitted** is authority (the agent's scoped policy). Addressed-but-not- permitted still demotes to room chatter, and the refusal wording still travels with the decision so the caller does not re-derive why. Tests follow the logic rather than being adapted to it: the mention-routing, system-marker and policy-enforcement suites now drive the resolver directly, which is where the behaviour they describe now lives. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…nt (CHOO-1436) Two event models, two dispatch entries and two no-op handlers, with no producer anywhere. Scoping the message log flagged them as never-sent and excluded them on that basis, which left an open question: either they are dead code, or something outside `switch_core` puts them on the bus — and the second would hole the premise that every message is somebody's outbound send, as well as meaning the denylist was dropping real traffic. They are dead, and have been since the initial import. No sender in `core/`, none in the three connectors, none in the console, none in the gateway; no generic "post an arbitrary event type" path an agent could reach the bus through; and no sender in the history to have been removed. On receipt they dispatched to handlers no subclass overrides, so an arriving one was silently dropped without even reaching the buffer. `NOT_SENT` goes with them. It named a category that no longer has members, and the denylist should not be documenting types the codebase does not know about. Note `docs/official/internals/matrix-substrate.md` still lists the pair. That tree is synced from the documentation repository and is not edited here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Rows exist from the moment the recorder was deployed. That was harmless while the read path went to the bus too; moving reads to Postgres turned it into a regression, and a large one — on the first deployment to record and then read, 731 of 739 bridged messages went from unqueried to invisible. Still on the homeserver, gone as far as any agent is concerned. `just backfill-messages` walks each room to its start and writes the messages that have no row. **Reconstructed rows are numbered below zero.** `seq` orders the room, so a message from last month cannot take a number above one from this morning, and it cannot take one among the live rows either — those are taken, and renumbering to make room would move every cursor pointing at them. Counting down from zero keeps the order right and `(room_id, seq)` unique, and makes the sign carry a fact worth reading off a row: positive was recorded as it happened, negative was reconstructed afterwards. The sign is load-bearing, not decorative. A delivery cursor starts at 0, so backfilling a year of history delivers none of it, which is the only sane answer — nobody wants last March pushed at them tonight. The notify trigger skips these rows for the same reason: every subscriber's cursor is above them by construction, so announcing them wakes the room to read nothing. **Idempotent, deliberately.** A walk over a busy room is long, and the useful thing to do with one that failed halfway is run it again. Each page commits on its own, `transport_event_id` is unique, and the walk checks before writing with the constraint as backstop. `--dry-run` walks exactly as far and makes exactly the same decisions, writing nothing. It reconstructs what the log is scoped to keep and no more, so it cannot put back what scoping the log took out. Arrivals come with it — membership is on the bus and the timeline reads wrong without them — carrying no body, so the sentence stays the reader's to phrase, exactly as the live path does. `sent_at` is the bus event's own timestamp, not the walk's clock. Getting that wrong would pile a year of history onto tonight and every time-windowed read would be wrong about all of it. Multi-file sends stay as they were sent: Matrix has no multi-attachment event, so coalescing would mean holding a page of parts and guessing which belong together. The group key survives in `content` for anyone who wants to later. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The notify listener has been built since #366 with no subscribers. This is its consumer: a transport watches each of its client's rooms, is woken when one advances, and reads the rows it has not seen into the same inbound events the Matrix transport produces. `since` is ignored, and that is behaviour to keep rather than an omission. A Matrix client resumed from its stored sync token and then discarded everything older than the process, so what it actually delivered on a restart was "whatever happened while I was up". Starting each room at its current head says that without the cursor that lied about it. What an agent missed while away is a delivery-cursor question, one layer up. Joining now writes the arrival as well as the membership. Over Matrix these were two things — the homeserver turned a join into an event, and Switch recorded that event separately — which is the split that let an arrival happen without being written. One write does both, and the room learns about a newcomer the same way it learns about anything else. The cursor advances per row, not per page: a handler that raises should cost its own event, not the ones delivered before it. Nothing constructs this transport yet.

… (CHOO-1436) (#368) * refactor(agent): call the eight RPC methods instead of posting events (CHOO-1436) Eight methods used the message bus as an RPC channel: register a future keyed on a request id, post a `com.switch.*` event into a room, block on `asyncio.wait_for`, and have a sync callback elsewhere complete the future. **Every one of them was switch-core talking to itself.** The responder was never out of process — for the two pre-invocation mediation calls the resource manager resolved the shared tracker in-process without even sending a reply, and for the other six a Switch-owned puppet received the event and answered. The homeserver was a loopback with a ten-second timeout bolted to it. What each actually was, once the round trip is removed: - `pre_tool_call` / `pre_llm_request` — one query against what the agent has attached. Now `MediationService`, which is also the first test coverage this logic has ever had; six cases, including that another agent's tool of the same name does not count. - `post_tool_result` / `post_llm_response` — **nothing.** Each posted an event carrying the literal string `"ok"` and read that same string back off the wire as its verdict. They are kept as the hook points they are meant to be, and as the membership check a caller is entitled to fail on, but the tautology is gone. Note their verdict vocabulary differs from the pre-invocation pair — `ok`/`blocked`/`redacted`, not `proceed`/`blocked` — which the round trip made easy to miss. - The four resource ones — `resource_service` calls. The gateway already called that service directly, so this is the existing shape, not a new one. Removing the hop removes several things that only existed to serve it: - Both trackers, which were the same forty lines twice over, differing in the future's value type and one log string. - `ResourceManagerClient` entirely. Once its six handlers go there is nothing left: it was a service wearing a Matrix client, and its only use of the room id was to map it back to the Switch room id the caller started with. It stops being a system client provisioned into every room. - Twelve event types, their models, their dispatch entries and their no-op base handlers — which empties the RPC bucket in `recorded_types.py`. - The sender-identity dance. Two of the eight sent as the resource manager rather than the agent, and it looked like routing. It was not: nothing dispatches on sender. It was a workaround for `_should_ignore` dropping a client's own events, so an agent sending its own response would have deadlocked until the timeout. Behaviour is preserved deliberately, including the parts that are not obviously right. A timeout used to surface as HTTP 404; there is no timeout now, and a real failure propagates with its type and traceback rather than being stringified into a `ValueError` — which is what the error-handling rules here ask for anyway. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(clients): delete the resource manager client row (CHOO-1436) The resource manager stopped being a Matrix client when the four resource operations became direct calls on `resource_service`, and nothing registers the type any more. Existing databases still carry the row, so `start_all` hands it to `ClientFactory.create`, which raises `Unknown client type: 'resource_manager'` and takes the whole process down before anything serves. Caught on the local deployment, where switch-core would not start at all. The row and its room memberships go; the Matrix account is left alone, since this migration owns the Switch database and not the homeserver. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(messages): classify retired types, group files, check arrivals (CHOO-1436) Four things the first live run of the step-6 stack turned up. **A deleted event type became permanent drift.** The denylist is built from the types the code still has, so the moment a type is deleted every historical event of it on the bus reads as unclassified — recorded by the rules, absent in fact, and reported as a missing row forever. The bus keeps its history for good, and step 7 deletes more types, in a tool that exits non-zero to gate a deploy. `RETIRED` names the thirteen types deleted so far and is append-only: writing needs today's types, walking history needs every type that ever existed, and only the first is allowed to shrink. **A two-file message read back as two messages.** The bus has no event that carries two files, so a multi-file post is n events sharing a group marker and n rows; live receivers coalesce them and the read path did not, so the second file came back as its own message captioned with a filename. It is reassembled on read, from the parts inside the window, led by the lowest index present. **An arrival was named after the platform.** The membership event's display name comes from the profile, which is set from whatever the source platform calls the member — so the same participant read one way arriving and another way speaking. The arriving client records its own name, the one its sends use. **Arrivals were the one write path nothing checked.** Reconciliation skipped membership on both sides, which made the newest path the unverified one. Joins are compared now, from the oldest recorded arrival onwards — every room was joined long before arrivals were recorded, so an older join cannot have a row and is counted rather than reported. A leave, an invite and a profile update re-firing as a join stay out of the comparison. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(agent): name an arrival the way Switch knows the member (CHOO-1436) A `room_join` event delivered live carried the name on the membership event, which is the Matrix profile, which a bridge sets from whatever the source platform calls someone — so the same arrival reached a connector as `charlie 💕` while the log recorded `charlie`. Two records of one event disagreeing about who it was about. The observing client resolves the arriving member through the client that owns their matrix id, which is where the recorded name comes from too. A matrix id no Switch client owns falls back to the membership event and says so in the log, rather than passing an empty name off as a resolution. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(provisioning): put a port in front of the homeserver admin (CHOO-1436) Step 2 gave the message path a port and left provisioning without one. `MatrixAdmin` is the second and last thing switch-core asks a homeserver for — accounts, rooms, membership — and five modules named the class directly, so removing Matrix meant editing every caller rather than supplying another implementation. `Provisioning` is derived from the six operations the call sites use, not from the endpoints behind them. `MatrixAdmin` satisfies it structurally and is unchanged; main.py still constructs it, as the composition root also names `MatrixTransport`. Membership stays invite/kick because that is what the operations mean. `invite_to_room` promises the user ends up in the room, which is what callers already rely on; an implementation that writes a row is free to. A Protocol's isinstance check compares names only, so the conformance test compares signatures too — a renamed parameter is exactly the drift the port exists to prevent and would otherwise type-check on both sides. * feat(transport): send over Postgres (CHOO-1436) The homeserver's remaining job is to carry an event from its sender to the clients that should see it and remember it in between. A table does all three, and Switch already writes that table beside every send. This makes the parallel record the thing itself: the write is the send. Consequences, stated in the module rather than discovered later: A send now fails when the database does. The recorder deliberately could not fail a send because a row was a nice-to-have next to a delivered message; here the row is the delivery, so there is nothing left to protect. Every durable event gets a row, not only the conversation — commands and task events crossed the bus too and have to reach their handlers. So `recorded_types` stops meaning "what is written" and starts meaning "what a reader is shown", applied on the way out. The exception is presence-like state, whose next value replaces it: announced, never stored. Receiving and media are not here. Both raise rather than returning something empty: a transport that silently delivers nothing looks like a healthy deployment with a silent room. Content assembly moves to `transport/content.py` so both transports build byte-identical events. Two copies would drift, and the drift would surface as a bridge rendering a caption on one deployment and a filename on another. Row shaping moves to `messages/row.py` for the same reason. `RoomStore.get_for_client` is the mirror of `get_client_ids`: over Matrix that question went to the homeserver, which answered it from the same memberships this table holds. * feat(transport): receive over Postgres (CHOO-1436) The notify listener has been built since #366 with no subscribers. This is its consumer: a transport watches each of its client's rooms, is woken when one advances, and reads the rows it has not seen into the same inbound events the Matrix transport produces. `since` is ignored, and that is behaviour to keep rather than an omission. A Matrix client resumed from its stored sync token and then discarded everything older than the process, so what it actually delivered on a restart was "whatever happened while I was up". Starting each room at its current head says that without the cursor that lied about it. What an agent missed while away is a delivery-cursor question, one layer up. Joining now writes the arrival as well as the membership. Over Matrix these were two things — the homeserver turned a join into an event, and Switch recorded that event separately — which is the split that let an arrival happen without being written. One write does both, and the room learns about a newcomer the same way it learns about anything else. The cursor advances per row, not per page: a handler that raises should cost its own event, not the ones delivered before it. Nothing constructs this transport yet. * feat(transport): store attachments in Postgres (CHOO-1436) The last thing the homeserver was holding. `media_blobs` takes the bytes and `upload_media` returns a key; what a message carries is unchanged in shape, because the handle was always opaque — it crosses the agent protocol as `mxc` and comes back as a query parameter, and nothing parses it. Object storage can replace the table later without the protocol noticing. `bytea` rather than a large object: attachments are capped at `agent_media_max_bytes` and are written and read whole, which is what TOAST is for and what large objects would only add lifecycle problems to. A handle with nothing behind it raises. Returning empty bytes would hand a reader a zero-byte file they cannot tell from a genuinely empty one, for an attachment the sender was told had been stored. Identical bytes uploaded twice are two rows. Deduplicating would make one sender's deletion another sender's data loss. Migration a1d7f3c95b60. Nothing reads the table until the transport is in use, so it can be applied ahead of the flip. * feat(provisioning): accounts, rooms and membership as rows (CHOO-1436) The second implementation of the port added three commits ago. Two of its six operations turn out to be almost nothing, and that is the finding rather than a shortcut: an account *is* the `clients` row the caller writes immediately afterwards, and a password is checked against that row rather than against a homeserver. Membership is the interesting one. Over Matrix an invitation was a durable event the invited client picked up whenever it next synced, so nothing had to know what order things happened in. A Postgres transport watches the rooms it knew about when it started, so a membership written underneath it is a room the client never reads — a client sitting in a room in silence. So a live client is woken and joins itself, through `InviteBus` and the same `on_invite` auto-accept the clients already have. When nobody is listening the membership is written directly: a client that is not running has nothing to wake and finds the room when it starts. The invitation is a wake-up, never the record. The bus is in-process, and the docstring says so. A client in another replica would not hear it — the same constraint Matrix sync sessions imposed, and the reason switch-core is single-replica today. Lifting it is its own step; until then this is no worse than what it replaces. No leave event is written on removal. A departure is not something a reader needs explained, and the timeline the log serves is what was said. `provisioning.py` becomes a package so the implementations sit beside the port, as the transport's do. Callers import the same name. * feat(transport): let a deployment run without a homeserver (CHOO-1436) `message_transport` picks matrix or postgres, and picks all of it: the transport, the provisioning implementation and whether a client records what it sent. Three settings would let a deployment choose a transport that stores what it carries *and* a recorder that writes it again, and both halves of that mistake are silent until someone reads the table. Defaults to matrix, so an existing deployment is unchanged by this commit. On postgres the homeserver is never contacted — not waited for, not logged into — because reaching a service the deployment does not use would make it a startup dependency of one that has replaced it. `MessageRecording` is the protocol behind that pairing. `NoRecording` is not a disabled feature: the rows still exist and are still one per event, written by the transport inside the transaction that accepted the send, which is stronger than the recorder it stands in for — that ran after delivery and could leave a gap. Known and disclosed at startup: media uploaded before the switch is served by the homeserver's store and cannot be fetched afterwards. Message history is unaffected — those rows are already in Postgres. * fix(bridges): put collaboration bridges on the chosen transport (CHOO-1436) The transport setting reached every agent and no bridge: the lifecycle service constructs BridgeClient itself and named matrix_transport_for directly, so a deployment on the Postgres transport ran agents on rows and four bridges on the homeserver. Neither side saw the other, nothing errored, and no message crossed to Slack or Mattermost. The choice is now made once, in ClientFactory, and the lifecycle service asks it — transport and recorder together, as everywhere else. A guard test refuses any module outside the factory naming a transport implementation at all; this call site has silently diverged from the factory twice, so the fix is removing the ability to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(bridges): record a bridge's membership of the rooms it carries (CHOO-1436) A bridge's rooms were expressed by inviting its client and letting the homeserver hold the membership, so nothing wrote them down. Once a client reads its rooms from client_rooms, a bridge had none: it kept receiving, because inbound posts into a room by id, and relayed nothing back out — healthy in every log, silent to everyone in Slack or Discord. Recorded at every bridge start rather than backfilled once, because the rooms a bridge carries change while it is stopped. Alongside it: a transport that starts receiving with no rooms says so at error level, since that state is otherwise indistinguishable from a quiet one. Not fatal — an agent with no rooms yet is normal. The transport guard test now forbids reaching the implementation modules rather than three symbol names, and walks plain imports as well as from-imports. A transport added later is covered when it lands instead of when someone remembers the list. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(transport): deliver presence that is announced and never stored (CHOO-1436) Runtime state was captured in agent_runtime_states and reached nobody: a Postgres transport delivers by reading rows, and the one event type that writes no row therefore had no delivery path. An agent going busy stopped showing up on a bridged channel while every log stayed clean. EPHEMERAL is unchanged — keeping presence out of the room's ordering was never the bug, only the delivery was. EphemeralBus carries the assembled event to the transports watching that room, so the announcement is the value rather than a position to read back. In-process, like InviteBus, and documented as such: both need the same cross-process form the day switch-core runs more than one replica, and they should get it together. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(rooms): make recording a membership idempotent, and always ring the bus (CHOO-1436) Over Matrix an invitation was accepted later, over sync, so room_service was effectively the only writer of client_rooms. On the Postgres transport the invitation is the join: the client writes the row synchronously and the inviting caller then writes it again. That raised an IntegrityError which rolled back the caller's transaction, so /invite-agent returned 500 and left an agent a client member of a room but not one of its agents — receiving messages while !list-agents reported nobody. The conflict is a correct outcome, not a failure, and tolerating it also closes the check-then-insert race between two concurrent invitations. invite_to_room now wakes the client before consulting the row. A row is no longer proof that a running client is subscribed to the room — something else may have written it, and there is a window at startup where a client is running but not yet on the bus. Returning early made that state permanent, because every later invitation found the same row and stopped. Membership is what must be exactly-once; a wake-up is cheap and coalesces. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(transport): give each client its own delivery loop (CHOO-1436) The listener fans out to every subscriber from a single task, and a waker awaited its handler all the way down to a bridge's HTTP call to Slack. One rate-limited bridge therefore stalled delivery for every client and every room in the process. Under Matrix each client had its own sync loop and got that isolation for free; collapsing N connections into one listener collapsed N delivery paths with them. A waker now notes the room and returns; each transport drains its own rooms on its own task. Serial within a client, because two concurrent runs for one room read the same cursor and deliver the same rows twice, and a redelivered message is indistinguishable from a new one. Overlapping wake-ups coalesce into one more pass, which also bounds a client to one outstanding read. join_room takes the subscription when the membership row already exists. Membership and subscription were one fact while the homeserver's join was what a sync loop delivered; they are two now, and returning early on the row left a client a member of a room it never reads, with no later join able to repair it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(transport): let a client hear its own arrival (CHOO-1436) An agent stopped greeting the room it was added to. The homeserver used to turn a join into an event and deliver it to everyone including the client it was about, down that client's own connection, and `on_self_join` fires on receiving it. Writing the row and then watching from the head steps over the client's own footprint: the room hears the arrival and the arriving client does not. The watch now starts just below the arrival it just wrote, so the join comes back through the ordinary delivery path with every guard downstream still in play. Restoring the loop-back rather than dispatching the one handler that noticed, because this is the third thing to break for the same reason today — Matrix told a client about its own actions, and several places quietly depended on that. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

…rgs (CHOO-1436) (#367) * fix(db): declare the four indexes only the migrations knew about (CHOO-1436) `ix_agent_sessions_agent_room`, `ix_agent_sessions_transport_session_id`, `ix_agents_parent_agent_id` and `ix_external_user_claims_user_id` were created by migrations and never declared on the models. Autogenerate would have emitted `DROP INDEX` for all four, so the next person to add a column had a one-keystroke path to quietly dropping four indexes off live tables. The parity test listed them by name rather than filtering by kind, precisely so this stayed visible. The list goes with them, and the test is back to allowing no drift at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(clients): type the constructor pass-through the bridges died on (CHOO-1436) Four `ClientBase` subclasses took their own arguments and forwarded the rest as `**kwargs: Any` (`**kwargs: object` with a `type: ignore` on `AgentClient`). That makes the pass-through invisible to the type checker on both sides: the subclass cannot be told it is missing something, and a caller cannot be told it is passing something that no longer exists. Which is how a stale `device_id=` type-checked clean and took all four collaboration bridges down at startup. The credential had moved into `session_state`, and nothing said so until the process refused to start. `ClientBaseKwargs` declares the shape once and the subclasses unpack it, so both checks come back without eleven parameters restated four times. Verified by putting the original mistake back: mypy now reports "Unexpected keyword argument "device_id" for "ClientBase"" at the call site. `matrix_transport_for` widens to `ClientBase[Any]`, which is what it always was in fact — it reads five base attributes and never the config. `ClientBase` is invariant in its config type, so the previous annotation only type-checked while nobody was checking. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(messages): drop the duplicate seq-paged read (CHOO-1436) `list_after_seq` was added for the delivery cursor without noticing that `list_for_room` already was that query, character for character. Two names for one behaviour is how they drift. `list_for_room` keeps the name and gains the paragraph that justified the second one, including the part worth writing down: a cursor starting at 0 also skips reconstructed history, which is numbered below zero, because a backfill is not something to deliver. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(agent): call the eight RPC methods instead of posting events (CHOO-1436) (#368) * refactor(agent): call the eight RPC methods instead of posting events (CHOO-1436) Eight methods used the message bus as an RPC channel: register a future keyed on a request id, post a `com.switch.*` event into a room, block on `asyncio.wait_for`, and have a sync callback elsewhere complete the future. **Every one of them was switch-core talking to itself.** The responder was never out of process — for the two pre-invocation mediation calls the resource manager resolved the shared tracker in-process without even sending a reply, and for the other six a Switch-owned puppet received the event and answered. The homeserver was a loopback with a ten-second timeout bolted to it. What each actually was, once the round trip is removed: - `pre_tool_call` / `pre_llm_request` — one query against what the agent has attached. Now `MediationService`, which is also the first test coverage this logic has ever had; six cases, including that another agent's tool of the same name does not count. - `post_tool_result` / `post_llm_response` — **nothing.** Each posted an event carrying the literal string `"ok"` and read that same string back off the wire as its verdict. They are kept as the hook points they are meant to be, and as the membership check a caller is entitled to fail on, but the tautology is gone. Note their verdict vocabulary differs from the pre-invocation pair — `ok`/`blocked`/`redacted`, not `proceed`/`blocked` — which the round trip made easy to miss. - The four resource ones — `resource_service` calls. The gateway already called that service directly, so this is the existing shape, not a new one. Removing the hop removes several things that only existed to serve it: - Both trackers, which were the same forty lines twice over, differing in the future's value type and one log string. - `ResourceManagerClient` entirely. Once its six handlers go there is nothing left: it was a service wearing a Matrix client, and its only use of the room id was to map it back to the Switch room id the caller started with. It stops being a system client provisioned into every room. - Twelve event types, their models, their dispatch entries and their no-op base handlers — which empties the RPC bucket in `recorded_types.py`. - The sender-identity dance. Two of the eight sent as the resource manager rather than the agent, and it looked like routing. It was not: nothing dispatches on sender. It was a workaround for `_should_ignore` dropping a client's own events, so an agent sending its own response would have deadlocked until the timeout. Behaviour is preserved deliberately, including the parts that are not obviously right. A timeout used to surface as HTTP 404; there is no timeout now, and a real failure propagates with its type and traceback rather than being stringified into a `ValueError` — which is what the error-handling rules here ask for anyway. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(clients): delete the resource manager client row (CHOO-1436) The resource manager stopped being a Matrix client when the four resource operations became direct calls on `resource_service`, and nothing registers the type any more. Existing databases still carry the row, so `start_all` hands it to `ClientFactory.create`, which raises `Unknown client type: 'resource_manager'` and takes the whole process down before anything serves. Caught on the local deployment, where switch-core would not start at all. The row and its room memberships go; the Matrix account is left alone, since this migration owns the Switch database and not the homeserver. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(messages): classify retired types, group files, check arrivals (CHOO-1436) Four things the first live run of the step-6 stack turned up. **A deleted event type became permanent drift.** The denylist is built from the types the code still has, so the moment a type is deleted every historical event of it on the bus reads as unclassified — recorded by the rules, absent in fact, and reported as a missing row forever. The bus keeps its history for good, and step 7 deletes more types, in a tool that exits non-zero to gate a deploy. `RETIRED` names the thirteen types deleted so far and is append-only: writing needs today's types, walking history needs every type that ever existed, and only the first is allowed to shrink. **A two-file message read back as two messages.** The bus has no event that carries two files, so a multi-file post is n events sharing a group marker and n rows; live receivers coalesce them and the read path did not, so the second file came back as its own message captioned with a filename. It is reassembled on read, from the parts inside the window, led by the lowest index present. **An arrival was named after the platform.** The membership event's display name comes from the profile, which is set from whatever the source platform calls the member — so the same participant read one way arriving and another way speaking. The arriving client records its own name, the one its sends use. **Arrivals were the one write path nothing checked.** Reconciliation skipped membership on both sides, which made the newest path the unverified one. Joins are compared now, from the oldest recorded arrival onwards — every room was joined long before arrivals were recorded, so an older join cannot have a row and is counted rather than reported. A leave, an invite and a profile update re-firing as a join stay out of the comparison. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(agent): name an arrival the way Switch knows the member (CHOO-1436) A `room_join` event delivered live carried the name on the membership event, which is the Matrix profile, which a bridge sets from whatever the source platform calls someone — so the same arrival reached a connector as `charlie 💕` while the log recorded `charlie`. Two records of one event disagreeing about who it was about. The observing client resolves the arriving member through the client that owns their matrix id, which is where the recorded name comes from too. A matrix id no Switch client owns falls back to the membership event and says so in the log, rather than passing an empty name off as a resolution. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(provisioning): put a port in front of the homeserver admin (CHOO-1436) Step 2 gave the message path a port and left provisioning without one. `MatrixAdmin` is the second and last thing switch-core asks a homeserver for — accounts, rooms, membership — and five modules named the class directly, so removing Matrix meant editing every caller rather than supplying another implementation. `Provisioning` is derived from the six operations the call sites use, not from the endpoints behind them. `MatrixAdmin` satisfies it structurally and is unchanged; main.py still constructs it, as the composition root also names `MatrixTransport`. Membership stays invite/kick because that is what the operations mean. `invite_to_room` promises the user ends up in the room, which is what callers already rely on; an implementation that writes a row is free to. A Protocol's isinstance check compares names only, so the conformance test compares signatures too — a renamed parameter is exactly the drift the port exists to prevent and would otherwise type-check on both sides. * feat(transport): send over Postgres (CHOO-1436) The homeserver's remaining job is to carry an event from its sender to the clients that should see it and remember it in between. A table does all three, and Switch already writes that table beside every send. This makes the parallel record the thing itself: the write is the send. Consequences, stated in the module rather than discovered later: A send now fails when the database does. The recorder deliberately could not fail a send because a row was a nice-to-have next to a delivered message; here the row is the delivery, so there is nothing left to protect. Every durable event gets a row, not only the conversation — commands and task events crossed the bus too and have to reach their handlers. So `recorded_types` stops meaning "what is written" and starts meaning "what a reader is shown", applied on the way out. The exception is presence-like state, whose next value replaces it: announced, never stored. Receiving and media are not here. Both raise rather than returning something empty: a transport that silently delivers nothing looks like a healthy deployment with a silent room. Content assembly moves to `transport/content.py` so both transports build byte-identical events. Two copies would drift, and the drift would surface as a bridge rendering a caption on one deployment and a filename on another. Row shaping moves to `messages/row.py` for the same reason. `RoomStore.get_for_client` is the mirror of `get_client_ids`: over Matrix that question went to the homeserver, which answered it from the same memberships this table holds. * feat(transport): receive over Postgres (CHOO-1436) The notify listener has been built since #366 with no subscribers. This is its consumer: a transport watches each of its client's rooms, is woken when one advances, and reads the rows it has not seen into the same inbound events the Matrix transport produces. `since` is ignored, and that is behaviour to keep rather than an omission. A Matrix client resumed from its stored sync token and then discarded everything older than the process, so what it actually delivered on a restart was "whatever happened while I was up". Starting each room at its current head says that without the cursor that lied about it. What an agent missed while away is a delivery-cursor question, one layer up. Joining now writes the arrival as well as the membership. Over Matrix these were two things — the homeserver turned a join into an event, and Switch recorded that event separately — which is the split that let an arrival happen without being written. One write does both, and the room learns about a newcomer the same way it learns about anything else. The cursor advances per row, not per page: a handler that raises should cost its own event, not the ones delivered before it. Nothing constructs this transport yet. * feat(transport): store attachments in Postgres (CHOO-1436) The last thing the homeserver was holding. `media_blobs` takes the bytes and `upload_media` returns a key; what a message carries is unchanged in shape, because the handle was always opaque — it crosses the agent protocol as `mxc` and comes back as a query parameter, and nothing parses it. Object storage can replace the table later without the protocol noticing. `bytea` rather than a large object: attachments are capped at `agent_media_max_bytes` and are written and read whole, which is what TOAST is for and what large objects would only add lifecycle problems to. A handle with nothing behind it raises. Returning empty bytes would hand a reader a zero-byte file they cannot tell from a genuinely empty one, for an attachment the sender was told had been stored. Identical bytes uploaded twice are two rows. Deduplicating would make one sender's deletion another sender's data loss. Migration a1d7f3c95b60. Nothing reads the table until the transport is in use, so it can be applied ahead of the flip. * feat(provisioning): accounts, rooms and membership as rows (CHOO-1436) The second implementation of the port added three commits ago. Two of its six operations turn out to be almost nothing, and that is the finding rather than a shortcut: an account *is* the `clients` row the caller writes immediately afterwards, and a password is checked against that row rather than against a homeserver. Membership is the interesting one. Over Matrix an invitation was a durable event the invited client picked up whenever it next synced, so nothing had to know what order things happened in. A Postgres transport watches the rooms it knew about when it started, so a membership written underneath it is a room the client never reads — a client sitting in a room in silence. So a live client is woken and joins itself, through `InviteBus` and the same `on_invite` auto-accept the clients already have. When nobody is listening the membership is written directly: a client that is not running has nothing to wake and finds the room when it starts. The invitation is a wake-up, never the record. The bus is in-process, and the docstring says so. A client in another replica would not hear it — the same constraint Matrix sync sessions imposed, and the reason switch-core is single-replica today. Lifting it is its own step; until then this is no worse than what it replaces. No leave event is written on removal. A departure is not something a reader needs explained, and the timeline the log serves is what was said. `provisioning.py` becomes a package so the implementations sit beside the port, as the transport's do. Callers import the same name. * feat(transport): let a deployment run without a homeserver (CHOO-1436) `message_transport` picks matrix or postgres, and picks all of it: the transport, the provisioning implementation and whether a client records what it sent. Three settings would let a deployment choose a transport that stores what it carries *and* a recorder that writes it again, and both halves of that mistake are silent until someone reads the table. Defaults to matrix, so an existing deployment is unchanged by this commit. On postgres the homeserver is never contacted — not waited for, not logged into — because reaching a service the deployment does not use would make it a startup dependency of one that has replaced it. `MessageRecording` is the protocol behind that pairing. `NoRecording` is not a disabled feature: the rows still exist and are still one per event, written by the transport inside the transaction that accepted the send, which is stronger than the recorder it stands in for — that ran after delivery and could leave a gap. Known and disclosed at startup: media uploaded before the switch is served by the homeserver's store and cannot be fetched afterwards. Message history is unaffected — those rows are already in Postgres. * fix(bridges): put collaboration bridges on the chosen transport (CHOO-1436) The transport setting reached every agent and no bridge: the lifecycle service constructs BridgeClient itself and named matrix_transport_for directly, so a deployment on the Postgres transport ran agents on rows and four bridges on the homeserver. Neither side saw the other, nothing errored, and no message crossed to Slack or Mattermost. The choice is now made once, in ClientFactory, and the lifecycle service asks it — transport and recorder together, as everywhere else. A guard test refuses any module outside the factory naming a transport implementation at all; this call site has silently diverged from the factory twice, so the fix is removing the ability to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(bridges): record a bridge's membership of the rooms it carries (CHOO-1436) A bridge's rooms were expressed by inviting its client and letting the homeserver hold the membership, so nothing wrote them down. Once a client reads its rooms from client_rooms, a bridge had none: it kept receiving, because inbound posts into a room by id, and relayed nothing back out — healthy in every log, silent to everyone in Slack or Discord. Recorded at every bridge start rather than backfilled once, because the rooms a bridge carries change while it is stopped. Alongside it: a transport that starts receiving with no rooms says so at error level, since that state is otherwise indistinguishable from a quiet one. Not fatal — an agent with no rooms yet is normal. The transport guard test now forbids reaching the implementation modules rather than three symbol names, and walks plain imports as well as from-imports. A transport added later is covered when it lands instead of when someone remembers the list. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(transport): deliver presence that is announced and never stored (CHOO-1436) Runtime state was captured in agent_runtime_states and reached nobody: a Postgres transport delivers by reading rows, and the one event type that writes no row therefore had no delivery path. An agent going busy stopped showing up on a bridged channel while every log stayed clean. EPHEMERAL is unchanged — keeping presence out of the room's ordering was never the bug, only the delivery was. EphemeralBus carries the assembled event to the transports watching that room, so the announcement is the value rather than a position to read back. In-process, like InviteBus, and documented as such: both need the same cross-process form the day switch-core runs more than one replica, and they should get it together. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(rooms): make recording a membership idempotent, and always ring the bus (CHOO-1436) Over Matrix an invitation was accepted later, over sync, so room_service was effectively the only writer of client_rooms. On the Postgres transport the invitation is the join: the client writes the row synchronously and the inviting caller then writes it again. That raised an IntegrityError which rolled back the caller's transaction, so /invite-agent returned 500 and left an agent a client member of a room but not one of its agents — receiving messages while !list-agents reported nobody. The conflict is a correct outcome, not a failure, and tolerating it also closes the check-then-insert race between two concurrent invitations. invite_to_room now wakes the client before consulting the row. A row is no longer proof that a running client is subscribed to the room — something else may have written it, and there is a window at startup where a client is running but not yet on the bus. Returning early made that state permanent, because every later invitation found the same row and stopped. Membership is what must be exactly-once; a wake-up is cheap and coalesces. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(transport): give each client its own delivery loop (CHOO-1436) The listener fans out to every subscriber from a single task, and a waker awaited its handler all the way down to a bridge's HTTP call to Slack. One rate-limited bridge therefore stalled delivery for every client and every room in the process. Under Matrix each client had its own sync loop and got that isolation for free; collapsing N connections into one listener collapsed N delivery paths with them. A waker now notes the room and returns; each transport drains its own rooms on its own task. Serial within a client, because two concurrent runs for one room read the same cursor and deliver the same rows twice, and a redelivered message is indistinguishable from a new one. Overlapping wake-ups coalesce into one more pass, which also bounds a client to one outstanding read. join_room takes the subscription when the membership row already exists. Membership and subscription were one fact while the homeserver's join was what a sync loop delivered; they are two now, and returning early on the row left a client a member of a room it never reads, with no later join able to repair it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(transport): let a client hear its own arrival (CHOO-1436) An agent stopped greeting the room it was added to. The homeserver used to turn a join into an event and deliver it to everyone including the client it was about, down that client's own connection, and `on_self_join` fires on receiving it. Writing the row and then watching from the head steps over the client's own footprint: the room hears the arrival and the arriving client does not. The watch now starts just below the arrival it just wrote, so the join comes back through the ordinary delivery path with every guard downstream still in play. Restoring the loop-back rather than dispatching the one handler that noticed, because this is the third thing to break for the same reason today — Matrix told a client about its own actions, and several places quietly depended on that. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

…verify it (CHOO-1436) (#365) * feat(db): record every message sent into a room (CHOO-1436) Adds `messages` and `message_attachments`, and a recorder that writes a row alongside each send. Nothing reads them yet. Every participant in a room is a Switch-owned client, so capturing sends captures the room exactly once — a bridged human's message enters through their puppet like any other. Capturing inbound as well would write N-1 duplicate rows per message in an N-client room. `seq` is a database-assigned total order. The read path will page on it: `sent_at` ties between messages sent in the same transaction, and a cursor with ties can skip or repeat rows across pages. Recording follows the send and cannot fail it. A database problem must not make messaging less reliable than it is today, so a failure leaves a gap and logs at error rather than raising into the caller. `delivery_cursors` is deliberately not added: nothing writes or reads it until the dispatcher exists, and its shape is that design's to decide. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(clients): route every send through the client so it can be recorded (CHOO-1436) `ClientBase` saw messages and media but not custom events: twenty call sites in the protocol service, the resource-manager client and the bridge reached past it to `transport.send_event`. They go through a new `ClientBase.send_event` instead, which is the layer that knows who is sending and can record it. That wrapper raises where `send_message` returns None. A chat message the homeserver rejects is a message nobody reads; a `com.switch.*` event that never lands is protocol state a caller is about to assume exists. `SendResult` now reports the event type and the content dict the transport put on the wire. The transport assembles that dict, so recording what was sent means transcribing it rather than rebuilding it — and a rebuild would drift from the transport the first time the wire format changed, silently. Tests cover each send route separately, since instrumenting one says nothing about the others, and cover the failure paths: a send the transport rejected records nothing, because a row for a message that never happened is worse than no row. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(messages): check the recorded messages against the message bus (CHOO-1436) Recording follows the send and cannot fail it, so a database problem leaves a row missing for a message that really was delivered. This measures how often that happens, which is the evidence the read path needs before it moves onto these rows. Unit tests cannot answer it: the question is whether two records of the same live traffic agree. Only the window in which recording was active is compared. Rows begin at the moment the recorder was deployed, so a room's whole back-catalogue is legitimately absent — reporting it would bury the one row that went astray. The window starts at the oldest recorded message unless the caller names an earlier point, and every report states which window it covered. The history walk is unbounded, unlike `read_context`, which stops at a page budget. A reconciliation that gave up early would report the events it never looked at as agreeing, which is the one answer it must never give. For the same reason a room the reader is not a member of is named and counted as a failure rather than skipped, and a room with nothing recorded is reported as having nothing to compare rather than as agreement. Fields are compared rather than the whole content dict: the bus may hand back an event annotated with fields it added itself, and reporting those as drift would train everyone to ignore the report. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(messages): record the conversation, not everything on the bus (CHOO-1436) Recording every custom event conflated two different things. The log is what a person reading the room later expects to find; the bus is everything a running system says to itself. A live check found ten of fourteen rows were runtime-state pings with a null body — a typing indicator, written synchronously on the send path, in the table the read path is about to query. Kept: `m.room.message` in both its text and media forms, and `com.switch.command`, which is a person typing `!invite-agent`. Everything else is bus traffic: ephemeral state, RPC pairs answered and forgotten, task transitions already durable in the `tasks` table, and per-run telemetry that would want a table shaped for querying it rather than a conversation log. Two declared permission types have no send site in the package at all. The classification is a denylist. An allowlist would drop a type nobody thought about, and a message the read path never knows to look for is the worse failure by far — so an unclassified type is recorded, and a test walks the dispatch table and names anything left undecided. Reconciliation follows the same rule. An unrecorded type is counted under its own name alongside the membership changes rather than reported as a missing row, because absent by design is not drift and a report full of it would be ignored. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(messages): number messages per room in commit order (CHOO-1436) `seq` was a global identity column, which is the obvious way to number rows and the wrong one for a cursor. A sequence allocates when the INSERT runs, not when the transaction commits. Two concurrent senders take 7 and 8; if 8 commits first, a reader paging on `seq > n` advances past it while 7 is still in flight, and when 7 lands it is already behind the cursor. The message was delivered, was recorded, and is never read. Rare, unreproducible and invisible — the worst shape of bug a message log could have, and the read path is about to page on exactly this column. The number is now assigned by the store under an advisory lock keyed on the room and held to commit, so writers to one room serialise and `seq` order and commit order become the same order. That is the only property a cursor needs. An advisory lock takes no row lock, so it cannot deadlock against anything that updates the room, and Postgres releases it on commit or rollback either way. Numbering is per room, so one busy room no longer pushes another room's first message to a large number, and the uniqueness constraint moved to (room_id, seq). The cost is per-room serialisation of one INSERT on a path that has already returned to its caller — recording begins after the send completed. Rooms do not contend with each other, and a test holds a transaction open to show it. Existing rows are renumbered within their room by their old global order, which is the order they were written in. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(messages): record arrivals in the message log (CHOO-1436) read_context returns two kinds of timeline entry — what someone said, and who arrived — and only the first was being recorded. Reading history from Postgres instead of the homeserver would therefore have silently dropped every join from the timeline, which is the part of the history that explains who is in the room and since when. A join is not a send, so it comes in by its own door: the arriving client records its own arrival. That keeps the property the send path gets for free — exactly one Switch client owns each participant, so exactly one of them writes the row, and there is nothing to deduplicate. The row is written on a genuine transition into membership, not under the guards that decide whether to announce the arrival. Those answer a different question: a join older than this process is nobody's news, but it is still how that participant got into the room. Seeing the same member event twice is not a second arrival, and the unique constraint on transport_event_id is the backstop behind the check that keeps that off the error log. No rendered sentence is stored. How an arrival reads is the reader's to phrase, and writing "x joined the room" into the log would freeze today's wording into every row ever written. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(messages): read room history from Postgres (CHOO-1436) read_context rebuilt the conversation by walking the homeserver backwards a page at a time. It now runs one windowed query against the message log, and both surfaces that expose history — the agent operation and GET /rooms/{id}/history — go through it, so there is one implementation and nothing to keep in step. What the walk needed, and what goes with it: - Two page budgets. Matrix has no timestamp cursor, so reaching a `before` deep in a busy room meant paging over everything newer; if that came out of the read budget a far-enough window returned empty no matter how small the limit, so seeking had to be budgeted apart from reading. A WHERE clause needs neither. - seek_by_timestamp, a hand-rolled call to timestamp_to_event with a manual bearer header because matrix-nio has no binding for it, plus a room_context round trip to turn the event id back into a pagination token. Deleted from the port, the Matrix transport and the test double. - A refetch per orphan thread root. Roots older than the window now come back with the page, in one query, scoped to the room so an id cannot be read across a room boundary by claiming it as a thread parent. `truncated` becomes exact. It was documented as deliberately conservative because the walk could not tell "the page cap stopped me" from "the room ended there"; the query asks for one row more than the caller wanted and the presence of that row is the answer. An arrival's sentence is composed on read rather than stored, so it can be reworded without rewriting history. Filtering a window needs an index on (room_id, sent_at): `seq` already has one through the room uniqueness constraint, and it is what the result is ordered by, but a time window had nothing to stand on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(messages): stop reconciliation reporting drift it invented (CHOO-1436) Run against a real deployment the checker reported 11 differences in a room that had none. Both causes were in the checker. The window began at the oldest row's `sent_at`. A row is written after its send is accepted, so that timestamp is always a little later than the event that produced it — the boundary excluded the very event it was derived from, and the row was left with nothing to match. Every room reports its first message as drift, on every run, for as long as the room exists. The boundary is now the oldest row's event id. Paging back to an identity rather than to a moment takes both clocks out of it, and the anchor is the one event admitted from before the window. The timestamp survives as a backstop so a room whose anchor is gone from the bus terminates rather than walking to the beginning of time; that outcome is reported as an unanchored window rather than as the difference it cannot account for. The second cause: the walk discards whole categories of event — arrivals, types the log deliberately does not keep — but the recorded side discarded nothing. A row of a discarded category therefore had nothing to be compared against and was reported as recorded-but-never-sent. Ten of the eleven were runtime-state rows written before the log was scoped, and they would have said the same thing forever. Both sides apply the filter now, and what it removes is counted under `unverifiable_by_type` — disclosed as outside the check rather than dropped from it. A checker that cries drift over its own arithmetic is worse than no checker, because it is meant to gate a deploy and this is exactly how it stops being read. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(messages): notify substrate, addressing extraction and history backfill (CHOO-1436) (#366) * feat(messages): let the message table announce its own inserts (CHOO-1436) The first half of moving delivery off Matrix. Nothing consumes this yet; it is the substrate the dispatcher will sit on. A trigger on `messages` fires `pg_notify` for every row, and a listener holds one `LISTEN` for the process's life and fans announcements out in-process to whoever asked about a room. **The announcement is a hint, never the payload.** It carries a room, a position and a row id; a subscriber reads the rows it has not seen. Under that rule everything awkward about Postgres's notification queue — not durable, not replayable, delivered at most once, and only to whoever happened to be connected — stops being a correctness problem, because the worst a lost announcement can do is delay a read the next one will trigger anyway. Putting the message in the payload would make the queue authoritative, and it is not built to be. Three consequences that are load-bearing rather than incidental: - Announcements coalesce per room. A burst during one handler run is one wake-up, and the handler still sees every row, because it works from its own cursor. That is also why there is no backpressure to apply and no queue to overflow. - Reconnecting wakes every subscriber. Announcements missed while disconnected are gone for good, so the listener treats a reconnect as "everything may have moved" rather than making each consumer detect its own gap. - It is a trigger rather than a call in the writer, so it cannot be forgotten by a future writer, and it commits with the row or not at all. A listener never hears about a row it could not then read. `delivery_cursors` lands with it — the table step 3 deferred. It persists what the event buffer held in memory, so an agent's position outlives the process serving it. `GREATEST` on advance, never an assignment: two deliveries can finish out of order, and rewinding a cursor redelivers. **Deviation from the plan: one channel, not one per room.** A per-room channel still needs its `LISTEN` issued before the first row it should catch, which is a race the consumer closes by reading the table — so the correctness machinery is identical either way and the subscription churn buys only fewer wakeups. The in-process fan-out does the routing instead. The listening connection is built outside the application pool. It is held, not borrowed; pool recycling would drop the subscription, and the silence afterwards is indistinguishable from a quiet room. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(delivery): decide addressing from stores, not from a client (CHOO-1436) Pure refactor, no behaviour change. It is what has to happen before delivery can be driven from the message table rather than from a Matrix sync loop. "Is this message for me?" lived on `AgentClient`, and every answer went through a live client — the agent came off `self`, the mxid off the transport session, the stores off the client's own wiring. A dispatcher reading rows out of `messages` has none of that, and the tempting shortcut is a second implementation for the second source. That is the one thing not to do. Two code paths answering "is this for me?" would be a security bug waiting for a rewrite to expose it: addressing is what the scoped addressing policy is enforced through, and a policy enforced by one path and not the other is worse than no policy. So the rules move to `switch_core/delivery/addressing.py` and take a message as data — a sender, a body, a content dict. `AgentClient` builds one of those from a bus event; the dispatcher will build one from a row; both get the same answer because it is the same code. The two questions stay apart, as they were: **addressed** is intent (name, room alias, a role held live, or the other party in a direct chat), and **permitted** is authority (the agent's scoped policy). Addressed-but-not- permitted still demotes to room chatter, and the refusal wording still travels with the decision so the caller does not re-derive why. Tests follow the logic rather than being adapted to it: the mention-routing, system-marker and policy-enforcement suites now drive the resolver directly, which is where the behaviour they describe now lives. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(events): delete the permission events, which nothing ever sent (CHOO-1436) Two event models, two dispatch entries and two no-op handlers, with no producer anywhere. Scoping the message log flagged them as never-sent and excluded them on that basis, which left an open question: either they are dead code, or something outside `switch_core` puts them on the bus — and the second would hole the premise that every message is somebody's outbound send, as well as meaning the denylist was dropping real traffic. They are dead, and have been since the initial import. No sender in `core/`, none in the three connectors, none in the console, none in the gateway; no generic "post an arbitrary event type" path an agent could reach the bus through; and no sender in the history to have been removed. On receipt they dispatched to handlers no subclass overrides, so an arriving one was silently dropped without even reaching the buffer. `NOT_SENT` goes with them. It named a category that no longer has members, and the denylist should not be documenting types the codebase does not know about. Note `docs/official/internals/matrix-substrate.md` still lists the pair. That tree is synced from the documentation repository and is not edited here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(messages): reconstruct room history from the bus (CHOO-1436) Rows exist from the moment the recorder was deployed. That was harmless while the read path went to the bus too; moving reads to Postgres turned it into a regression, and a large one — on the first deployment to record and then read, 731 of 739 bridged messages went from unqueried to invisible. Still on the homeserver, gone as far as any agent is concerned. `just backfill-messages` walks each room to its start and writes the messages that have no row. **Reconstructed rows are numbered below zero.** `seq` orders the room, so a message from last month cannot take a number above one from this morning, and it cannot take one among the live rows either — those are taken, and renumbering to make room would move every cursor pointing at them. Counting down from zero keeps the order right and `(room_id, seq)` unique, and makes the sign carry a fact worth reading off a row: positive was recorded as it happened, negative was reconstructed afterwards. The sign is load-bearing, not decorative. A delivery cursor starts at 0, so backfilling a year of history delivers none of it, which is the only sane answer — nobody wants last March pushed at them tonight. The notify trigger skips these rows for the same reason: every subscriber's cursor is above them by construction, so announcing them wakes the room to read nothing. **Idempotent, deliberately.** A walk over a busy room is long, and the useful thing to do with one that failed halfway is run it again. Each page commits on its own, `transport_event_id` is unique, and the walk checks before writing with the constraint as backstop. `--dry-run` walks exactly as far and makes exactly the same decisions, writing nothing. It reconstructs what the log is scoped to keep and no more, so it cannot put back what scoping the log took out. Arrivals come with it — membership is on the bus and the timeline reads wrong without them — carrying no body, so the sentence stays the reader's to phrase, exactly as the live path does. `sent_at` is the bus event's own timestamp, not the walk's clock. Getting that wrong would pile a year of history onto tonight and every time-windowed read would be wrong about all of it. Multi-file sends stay as they were sent: Matrix has no multi-attachment event, so coalescing would mean holding a page of parts and guessing which belong together. The group key survives in `content` for anyone who wants to later. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(db,clients): declare the undeclared indexes, type the untyped kwargs (CHOO-1436) (#367) * fix(db): declare the four indexes only the migrations knew about (CHOO-1436) `ix_agent_sessions_agent_room`, `ix_agent_sessions_transport_session_id`, `ix_agents_parent_agent_id` and `ix_external_user_claims_user_id` were created by migrations and never declared on the models. Autogenerate would have emitted `DROP INDEX` for all four, so the next person to add a column had a one-keystroke path to quietly dropping four indexes off live tables. The parity test listed them by name rather than filtering by kind, precisely so this stayed visible. The list goes with them, and the test is back to allowing no drift at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(clients): type the constructor pass-through the bridges died on (CHOO-1436) Four `ClientBase` subclasses took their own arguments and forwarded the rest as `**kwargs: Any` (`**kwargs: object` with a `type: ignore` on `AgentClient`). That makes the pass-through invisible to the type checker on both sides: the subclass cannot be told it is missing something, and a caller cannot be told it is passing something that no longer exists. Which is how a stale `device_id=` type-checked clean and took all four collaboration bridges down at startup. The credential had moved into `session_state`, and nothing said so until the process refused to start. `ClientBaseKwargs` declares the shape once and the subclasses unpack it, so both checks come back without eleven parameters restated four times. Verified by putting the original mistake back: mypy now reports "Unexpected keyword argument "device_id" for "ClientBase"" at the call site. `matrix_transport_for` widens to `ClientBase[Any]`, which is what it always was in fact — it reads five base attributes and never the config. `ClientBase` is invariant in its config type, so the previous annotation only type-checked while nobody was checking. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(messages): drop the duplicate seq-paged read (CHOO-1436) `list_after_seq` was added for the delivery cursor without noticing that `list_for_room` already was that query, character for character. Two names for one behaviour is how they drift. `list_for_room` keeps the name and gains the paragraph that justified the second one, including the part worth writing down: a cursor starting at 0 also skips reconstructed history, which is numbered below zero, because a backfill is not something to deliver. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(agent): call the eight RPC methods instead of posting events (CHOO-1436) (#368) * refactor(agent): call the eight RPC methods instead of posting events (CHOO-1436) Eight methods used the message bus as an RPC channel: register a future keyed on a request id, post a `com.switch.*` event into a room, block on `asyncio.wait_for`, and have a sync callback elsewhere complete the future. **Every one of them was switch-core talking to itself.** The responder was never out of process — for the two pre-invocation mediation calls the resource manager resolved the shared tracker in-process without even sending a reply, and for the other six a Switch-owned puppet received the event and answered. The homeserver was a loopback with a ten-second timeout bolted to it. What each actually was, once the round trip is removed: - `pre_tool_call` / `pre_llm_request` — one query against what the agent has attached. Now `MediationService`, which is also the first test coverage this logic has ever had; six cases, including that another agent's tool of the same name does not count. - `post_tool_result` / `post_llm_response` — **nothing.** Each posted an event carrying the literal string `"ok"` and read that same string back off the wire as its verdict. They are kept as the hook points they are meant to be, and as the membership check a caller is entitled to fail on, but the tautology is gone. Note their verdict vocabulary differs from the pre-invocation pair — `ok`/`blocked`/`redacted`, not `proceed`/`blocked` — which the round trip made easy to miss. - The four resource ones — `resource_service` calls. The gateway already called that service directly, so this is the existing shape, not a new one. Removing the hop removes several things that only existed to serve it: - Both trackers, which were the same forty lines twice over, differing in the future's value type and one log string. - `ResourceManagerClient` entirely. Once its six handlers go there is nothing left: it was a service wearing a Matrix client, and its only use of the room id was to map it back to the Switch room id the caller started with. It stops being a system client provisioned into every room. - Twelve event types, their models, their dispatch entries and their no-op base handlers — which empties the RPC bucket in `recorded_types.py`. - The sender-identity dance. Two of the eight sent as the resource manager rather than the agent, and it looked like routing. It was not: nothing dispatches on sender. It was a workaround for `_should_ignore` dropping a client's own events, so an agent sending its own response would have deadlocked until the timeout. Behaviour is preserved deliberately, including the parts that are not obviously right. A timeout used to surface as HTTP 404; there is no timeout now, and a real failure propagates with its type and traceback rather than being stringified into a `ValueError` — which is what the error-handling rules here ask for anyway. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(clients): delete the resource manager client row (CHOO-1436) The resource manager stopped being a Matrix client when the four resource operations became direct calls on `resource_service`, and nothing registers the type any more. Existing databases still carry the row, so `start_all` hands it to `ClientFactory.create`, which raises `Unknown client type: 'resource_manager'` and takes the whole process down before anything serves. Caught on the local deployment, where switch-core would not start at all. The row and its room memberships go; the Matrix account is left alone, since this migration owns the Switch database and not the homeserver. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(messages): classify retired types, group files, check arrivals (CHOO-1436) Four things the first live run of the step-6 stack turned up. **A deleted event type became permanent drift.** The denylist is built from the types the code still has, so the moment a type is deleted every historical event of it on the bus reads as unclassified — recorded by the rules, absent in fact, and reported as a missing row forever. The bus keeps its history for good, and step 7 deletes more types, in a tool that exits non-zero to gate a deploy. `RETIRED` names the thirteen types deleted so far and is append-only: writing needs today's types, walking history needs every type that ever existed, and only the first is allowed to shrink. **A two-file message read back as two messages.** The bus has no event that carries two files, so a multi-file post is n events sharing a group marker and n rows; live receivers coalesce them and the read path did not, so the second file came back as its own message captioned with a filename. It is reassembled on read, from the parts inside the window, led by the lowest index present. **An arrival was named after the platform.** The membership event's display name comes from the profile, which is set from whatever the source platform calls the member — so the same participant read one way arriving and another way speaking. The arriving client records its own name, the one its sends use. **Arrivals were the one write path nothing checked.** Reconciliation skipped membership on both sides, which made the newest path the unverified one. Joins are compared now, from the oldest recorded arrival onwards — every room was joined long before arrivals were recorded, so an older join cannot have a row and is counted rather than reported. A leave, an invite and a profile update re-firing as a join stay out of the comparison. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(agent): name an arrival the way Switch knows the member (CHOO-1436) A `room_join` event delivered live carried the name on the membership event, which is the Matrix profile, which a bridge sets from whatever the source platform calls someone — so the same arrival reached a connector as `charlie 💕` while the log recorded `charlie`. Two records of one event disagreeing about who it was about. The observing client resolves the arriving member through the client that owns their matrix id, which is where the recorded name comes from too. A matrix id no Switch client owns falls back to the membership event and says so in the log, rather than passing an empty name off as a resolution. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(provisioning): put a port in front of the homeserver admin (CHOO-1436) Step 2 gave the message path a port and left provisioning without one. `MatrixAdmin` is the second and last thing switch-core asks a homeserver for — accounts, rooms, membership — and five modules named the class directly, so removing Matrix meant editing every caller rather than supplying another implementation. `Provisioning` is derived from the six operations the call sites use, not from the endpoints behind them. `MatrixAdmin` satisfies it structurally and is unchanged; main.py still constructs it, as the composition root also names `MatrixTransport`. Membership stays invite/kick because that is what the operations mean. `invite_to_room` promises the user ends up in the room, which is what callers already rely on; an implementation that writes a row is free to. A Protocol's isinstance check compares names only, so the conformance test compares signatures too — a renamed parameter is exactly the drift the port exists to prevent and would otherwise type-check on both sides. * feat(transport): send over Postgres (CHOO-1436) The homeserver's remaining job is to carry an event from its sender to the clients that should see it and remember it in between. A table does all three, and Switch already writes that table beside every send. This makes the parallel record the thing itself: the write is the send. Consequences, stated in the module rather than discovered later: A send now fails when the database does. The recorder deliberately could not fail a send because a row was a nice-to-have next to a delivered message; here the row is the delivery, so there is nothing left to protect. Every durable event gets a row, not only the conversation — commands and task events crossed the bus too and have to reach their handlers. So `recorded_types` stops meaning "what is written" and starts meaning "what a reader is shown", applied on the way out. The exception is presence-like state, whose next value replaces it: announced, never stored. Receiving and media are not here. Both raise rather than returning something empty: a transport that silently delivers nothing looks like a healthy deployment with a silent room. Content assembly moves to `transport/content.py` so both transports build byte-identical events. Two copies would drift, and the drift would surface as a bridge rendering a caption on one deployment and a filename on another. Row shaping moves to `messages/row.py` for the same reason. `RoomStore.get_for_client` is the mirror of `get_client_ids`: over Matrix that question went to the homeserver, which answered it from the same memberships this table holds. * feat(transport): receive over Postgres (CHOO-1436) The notify listener has been built since #366 with no subscribers. This is its consumer: a transport watches each of its client's rooms, is woken when one advances, and reads the rows it has not seen into the same inbound events the Matrix transport produces. `since` is ignored, and that is behaviour to keep rather than an omission. A Matrix client resumed from its stored sync token and then discarded everything older than the process, so what it actually delivered on a restart was "whatever happened while I was up". Starting each room at its current head says that without the cursor that lied about it. What an agent missed while away is a delivery-cursor question, one layer up. Joining now writes the arrival as well as the membership. Over Matrix these were two things — the homeserver turned a join into an event, and Switch recorded that event separately — which is the split that let an arrival happen without being written. One write does both, and the room learns about a newcomer the same way it learns about anything else. The cursor advances per row, not per page: a handler that raises should cost its own event, not the ones delivered before it. Nothing constructs this transport yet. * feat(transport): store attachments in Postgres (CHOO-1436) The last thing the homeserver was holding. `media_blobs` takes the bytes and `upload_media` returns a key; what a message carries is unchanged in shape, because the handle was always opaque — it crosses the agent protocol as `mxc` and comes back as a query parameter, and nothing parses it. Object storage can replace the table later without the protocol noticing. `bytea` rather than a large object: attachments are capped at `agent_media_max_bytes` and are written and read whole, which is what TOAST is for and what large objects would only add lifecycle problems to. A handle with nothing behind it raises. Returning empty bytes would hand a reader a zero-byte file they cannot tell from a genuinely empty one, for an attachment the sender was told had been stored. Identical bytes uploaded twice are two rows. Deduplicating would make one sender's deletion another sender's data loss. Migration a1d7f3c95b60. Nothing reads the table until the transport is in use, so it can be applied ahead of the flip. * feat(provisioning): accounts, rooms and membership as rows (CHOO-1436) The second implementation of the port added three commits ago. Two of its six operations turn out to be almost nothing, and that is the finding rather than a shortcut: an account *is* the `clients` row the caller writes immediately afterwards, and a password is checked against that row rather than against a homeserver. Membership is the interesting one. Over Matrix an invitation was a durable event the invited client picked up whenever it next synced, so nothing had to know what order things happened in. A Postgres transport watches the rooms it knew about when it started, so a membership written underneath it is a room the client never reads — a client sitting in a room in silence. So a live client is woken and joins itself, through `InviteBus` and the same `on_invite` auto-accept the clients already have. When nobody is listening the membership is written directly: a client that is not running has nothing to wake and finds the room when it starts. The invitation is a wake-up, never the record. The bus is in-process, and the docstring says so. A client in another replica would not hear it — the same constraint Matrix sync sessions imposed, and the reason switch-core is single-replica today. Lifting it is its own step; until then this is no worse than what it replaces. No leave event is written on removal. A departure is not something a reader needs explained, and the timeline the log serves is what was said. `provisioning.py` becomes a package so the implementations sit beside the port, as the transport's do. Callers import the same name. * feat(transport): let a deployment run without a homeserver (CHOO-1436) `message_transport` picks matrix or postgres, and picks all of it: the transport, the provisioning implementation and whether a client records what it sent. Three settings would let a deployment choose a transport that stores what it carries *and* a recorder that writes it again, and both halves of that mistake are silent until someone reads the table. Defaults to matrix, so an existing deployment is unchanged by this commit. On postgres the homeserver is never contacted — not waited for, not logged into — because reaching a service the deployment does not use would make it a startup dependency of one that has replaced it. `MessageRecording` is the protocol behind that pairing. `NoRecording` is not a disabled feature: the rows still exist and are still one per event, written by the transport inside the transaction that accepted the send, which is stronger than the recorder it stands in for — that ran after delivery and could leave a gap. Known and disclosed at startup: media uploaded before the switch is served by the homeserver's store and cannot be fetched afterwards. Message history is unaffected — those rows are already in Postgres. * fix(bridges): put collaboration bridges on the chosen transport (CHOO-1436) The transport setting reached every agent and no bridge: the lifecycle service constructs BridgeClient itself and named matrix_transport_for directly, so a deployment on the Postgres transport ran agents on rows and four bridges on the homeserver. Neither side saw the other, nothing errored, and no message crossed to Slack or Mattermost. The choice is now made once, in ClientFactory, and the lifecycle service asks it — transport and recorder together, as everywhere else. A guard test refuses any module outside the factory naming a transport implementation at all; this call site has silently diverged from the factory twice, so the fix is removing the ability to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(bridges): record a bridge's membership of the rooms it carries (CHOO-1436) A bridge's rooms were expressed by inviting its client and letting the homeserver hold the membership, so nothing wrote them down. Once a client reads its rooms from client_rooms, a bridge had none: it kept receiving, because inbound posts into a room by id, and relayed nothing back out — healthy in every log, silent to everyone in Slack or Discord. Recorded at every bridge start rather than backfilled once, because the rooms a bridge carries change while it is stopped. Alongside it: a transport that starts receiving with no rooms says so at error level, since that state is otherwise indistinguishable from a quiet one. Not fatal — an agent with no rooms yet is normal. The transport guard test now forbids reaching the implementation modules rather than three symbol names, and walks plain imports as well as from-imports. A transport added later is covered when it lands instead of when someone remembers the list. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(transport): deliver presence that is announced and never stored (CHOO-1436) Runtime state was captured in agent_runtime_states and reached nobody: a Postgres transport delivers by reading rows, and the one event type that writes no row therefore had no delivery path. An agent going busy stopped showing up on a bridged channel while every log stayed clean. EPHEMERAL is unchanged — keeping presence out of the room's ordering was never the bug, only the delivery was. EphemeralBus carries the assembled event to the transports watching that room, so the announcement is the value rather than a position to read back. In-process, like InviteBus, and documented as such: both need the same cross-process form the day switch-core runs more than one replica, and they should get it together. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(rooms): make recording a membership idempotent, and always ring the bus (CHOO-1436) Over Matrix an invitation was accepted later, over sync, so room_service was effectively the only writer of client_rooms. On the Postgres transport the invitation is the join: the client writes the row synchronously and the inviting caller then writes it again. That raised an IntegrityError which rolled back the caller's transaction, so /invite-agent returned 500 and left an agent a client member of a room but not one of its agents — receiving messages while !list-agents reported nobody. The conflict is a correct outcome, not a failure, and tolerating it also closes the check-then-insert race between two concurrent invitations. invite_to_room now wakes the client before consulting the row. A row is no longer proof that a running client is subscribed to the room — something else may have written it, and there is a window at startup where a client is running but not yet on the bus. Returning early made that state permanent, because every later invitation found the same row and stopped. Membership is what must be exactly-once; a wake-up is cheap and coalesces. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(transport): give each client its own delivery loop (CHOO-1436) The listener fans out to every subscriber from a single task, and a waker awaited its handler all the way down to a bridge's HTTP call to Slack. One rate-limited bridge therefore stalled delivery for every client and every room in the process. Under Matrix each client had its own sync loop and got that isolation for free; collapsing N connections into one listener collapsed N delivery paths with them. A waker now notes the room and returns; each transport drains its own rooms on its own task. Serial within a client, because two concurrent runs for one room read the same cursor and deliver the same rows twice, and a redelivered message is indistinguishable from a new one. Overlapping wake-ups coalesce into one more pass, which also bounds a client to one outstanding read. join_room takes the subscription when the membership row already exists. Membership and subscription were one fact while the homeserver's join was what a sync loop delivered; they are two now, and returning early on the row left a client a member of a room it never reads, with no later join able to repair it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(transport): let a client hear its own arrival (CHOO-1436) An agent stopped greeting the room it was added to. The homeserver used to turn a join into an event and deliver it to everyone including the client it was about, down that client's own connection, and `on_self_join` fires on receiving it. Writing the row and then watching from the head steps over the client's own footprint: the room hears the arrival and the arriving client does not. The watch now starts just below the arrival it just wrote, so the join comes back through the ordinary delivery path with every guard downstream still in play. Restoring the loop-back rather than dispatching the one handler that noticed, because this is the third thing to break for the same reason today — Matrix told a client about its own actions, and several places quietly depended on that. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

amaudruz and others added 4 commits September 3, 2026 21:56

amaudruz changed the title ~~feat(messages): let the message table announce its own inserts (CHOO-1436)~~ feat(messages): notify substrate, addressing extraction and history backfill (CHOO-1436) Sep 3, 2026

amaudruz mentioned this pull request Sep 3, 2026

fix(db,clients): declare the undeclared indexes, type the untyped kwargs (CHOO-1436) #367

Merged

amaudruz marked this pull request as ready for review September 4, 2026 14:31

amaudruz requested a review from christian-mcdermott as a code owner September 4, 2026 14:31

amaudruz merged commit 582f628 into worktree-message-store Sep 4, 2026
9 of 10 checks passed

Navigation Menu

Uh oh!

feat(messages): notify substrate, addressing extraction and history backfill (CHOO-1436) - #366

feat(messages): notify substrate, addressing extraction and history backfill (CHOO-1436)#366
amaudruz merged 5 commits into
worktree-message-storefrom
worktree-message-notify

amaudruz commented Sep 3, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

amaudruz commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

1. The notify substrate — 857807af

2. Addressing decided from stores, not from a client — 3e0f0162

3. The permission events, deleted — 9cbf173e (outstanding #3)

4. History backfill — cd4b38a2 (outstanding #1)

What is NOT here, and why

Testing

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

amaudruz commented Sep 3, 2026 •

edited

Loading

1. The notify substrate — `857807af`

2. Addressing decided from stores, not from a client — `3e0f0162`

3. The permission events, deleted — `9cbf173e` (outstanding #3)

4. History backfill — `cd4b38a2` (outstanding #1)