Attach reliability: failure-mode map for every disconnect path (drive/view/passthrough, local and --node)

[khaliqgant]

Summary

agent-relay node agent attach disconnects, flaps, or reports alarming errors during normal use. This issue maps every disconnect path in the attach stack: what triggers it, whether it self-heals, and what the operator sees. It is the prerequisite for the fixes proposed at the end.

Scope: packages/cli/src/cli/lib/attach*.ts (18 files), packages/harness-driver/src/transport.ts, crates/broker/src/listen_api.rs, crates/broker/src/pty_worker.rs, crates/relay-pty/src/pty.rs.

Headline answer to "is the input stream lost / reconnected pair benign?" The session does survive — but it is not benign, and it is not a cosmetic wording problem. The broker really closed the socket, the keystroke that triggered it was dropped, and the cause persists, so it recurs on the next keystroke for as long as the agent stays busy. The genuinely dangerous part is item 3 below: input already accepted by the broker is not dropped; it is replayed into the agent later, out of context.

---

Failure-mode map

1. Broker closes the PTY input WebSocket on any write error except worker_timeout

• crates/broker/src/listen_api.rs:1788 — pty_input_error_is_connection_fatal(code) = code != "worker_timeout", applied at listen_api.rs:1896.
• Trigger: the worker returns pty_write_failed (crates/broker/src/pty_worker.rs:993), most commonly because relay-pty's bounded write queue is full — WRITE_QUEUE_DEPTH = 128 at crates/relay-pty/src/pty.rs:31, message emitted at crates/relay-pty/src/pty.rs:502.
• Self-heals: yes — the client reopens the stream (packages/cli/src/cli/lib/attach-input-recovery.ts:311, identity-gated). But the cause is unaffected by reconnecting, so the next keystroke reproduces it.
• Operator sees: [drive] input stream lost (pty write queue full (128 writes pending; drainer wedged behind child not reading stdin)); reconnecting… followed by reconnected after 1 attempt(s). Indistinguishable from a fatal error.
• Cost: the triggering keystroke is dropped and its optimistic echo is rolled back.
• Verified by code path. The 128 cap itself was not reproduced live (see E2/E3).

2. The 128-slot queue is shared, so a human is punished for writes they did not make

• crates/relay-pty/src/pty.rs:20-30: the same queue carries alacritty terminal-query replies (DSR/DA1/DA2/CPR), relay message injections, harness auto-responder writes, and human keystrokes.
• Human keystrokes are serialized per agent (listen_api.rs:2027) and each waits up to PTY_INPUT_ACK_TIMEOUT = 5s (crates/broker/src/runtime/api.rs:28), so a human can add at most ~1 queue entry per 5 s. A human cannot fill 128 slots by typing.
• Therefore a queue-full error seen by an operator is essentially always caused by other writers — a busy TUI answering query sequences, or injected messages.

3. Accepted input is never dropped — it is replayed into the agent later ⚠️

• The CLI correctly drops its own keystrokes during an outage rather than buffering them (packages/cli/src/cli/lib/attach-drive.ts stdin handler).
• But anything already accepted into the broker's drainer queue stays there and is written to the child whenever it resumes reading — with unbounded delay.
• Verified (E4): after SIGCONT, agent-relay-broker dump-pty showed keystrokes typed minutes earlier and injected message bodies executing as shell commands in the agent.
• So the answer to "does reconnect drop or replay queued input" is: the client drops, the broker replays. Stale input does reach a live agent.

4. The events/screen WebSocket has no reconnect at all

• packages/cli/src/cli/lib/attach-drive.ts:1445-1457: any error → finish(1); any close code other than 1000/1005 → finish(1). attach-view.ts / attach-passthrough.ts follow the same shape.
• No resume is attempted even though the stream URL already supports ?sinceSeq= (attach-drive.ts:740) and a StreamSyncBuffer already reconciles replayed chunks.
• Self-heals: no. One transient blip ends the session.
• Operator sees: [drive] connection closed (code: 1006) and an exit.
• This is the most likely explanation for "I notice when attached it sometimes disconnects".

5. The PTY input WebSocket has no keepalive in either direction

• The broker pings the events WS every 30 s (listen_api.rs:2999, listen_api.rs:3049). handle_pty_input_ws only answers pings (listen_api.rs:2098) and never sends one; the client never pings either (packages/harness-driver/src/transport.ts has no ping path).
• An idle input socket is therefore silent on the wire and dies to any intermediary idle timeout (cloud edge on --node attaches, NAT, proxy) while the screen keeps updating — a seat that looks alive with dead input until the next keystroke.
• This is documented as a known hazard in attach-input-recovery.ts:1-25; the response was client-side recovery, never a transport-level keepalive.

6. Degenerate terminal size is forwarded verbatim as a resize

• packages/cli/src/cli/lib/attach.ts:519-528 — reserveStatusLineRow returns the size unchanged when rows < 3 || cols < 2, so a 0x0 TTY (e.g. script(1) with no winsize, a minimized window, a size not yet reported) is sent as rows: 0, cols: 0, and is rejected by attach-fleet-node.ts:675 / pty_worker.rs:936 as invalid_dimensions.
• Operator sees: [drive] could not sync agent PTY size to local terminal (rows and cols must be positive integers); continuing.
• Non-fatal — verified, the session continues — but this string has already sent two separate investigations down the wrong path.

7. Cross-node agent without --node returns a near-empty stream instead of an error

• Attaching to an agent that lives on another node without --node produced ~54 bytes of stream rather than an error naming the node. The fleet placement hint (packages/cli/src/cli/lib/fleet-hint.ts) only fires on a 404 from the delivery-mode call, so a name the local broker resolves but cannot render never reaches it.
• Not yet root-caused. Silent wrong answer.

8. Age-dependent behaviour (relay#1593) — unverified

Attach to agents older than ~5 h reportedly behaves differently from fresh ones. Not reproduced here; noted so it is not lost.

---

Experiments (evidence)

Run against a throwaway sh agent (wedge-probe-0821) on the local broker, driven from a real PTY (40×120) by a Python pty.fork() harness, stdout and stderr captured separately. The flagship lead was never driven.

• E1 — drive attach to a live agent: works; status line renders; keystrokes land.
• E2 — child SIGSTOPped (a faithful stand-in for a harness that stops reading stdin mid-tool-call) + 2500 keystrokes: exactly one line, [drive] <agent> did not confirm a keystroke in time (worker busy); it may still land. Session survived, no reconnect. The relay#1544 handling works as designed.
• E3 — 150 --mode steer DMs injected into the wedged agent, then typing: same benign path; the 128 cap was still not reached.
• E4 — SIGCONT, then dump-pty: keystrokes and injected message bodies from minutes earlier executed as shell commands. This is item 3.
• OS note — on macOS a pty master write blocks once the slave's input queue fills (measured: ~768 bytes with the slave in raw mode) but discards in canonical mode. So the wedge requires a child whose tty is in raw mode — i.e. exactly a TUI harness, which is why this is only seen against real agents.

---

Proposal, ranked by operator impact per line changed

1. A full write queue must not kill the input stream. Give queue-full a distinct code end to end (relay-pty → pty_worker → listen_api) and exempt only that code in pty_input_error_is_connection_fatal, exactly as worker_timeout is exempt. The worker is provably alive — it refused one write. This turns the entire flap in item 1 into a single non-fatal line. Highest leverage in the stack.
2. Keepalive on the PTY input WS (client-side ping, ~20 s). Closes item 5 for every broker version, since the broker already answers pings. Removes the silent "screen alive, input dead" seat.
3. Truthful operator messaging. A recovered flap and a fatal disconnect must not read alike: on recovery say the session is usable and what was lost; on a real disconnect keep the loud message. Closes the design constraint and item 1's readability half.
4. Make backpressure visible before the cap, not after it wedges: report the drainer queue depth on the write ack and surface a status-line indicator when the agent stops consuming input.
5. Clamp degenerate sizes in reserveStatusLineRow (item 6) — a few lines, removes a misleading error string permanently.
6. Events-WS resume via sinceSeq (item 4). The biggest operator win for "stay attached", but also the biggest change; proposed as a follow-up rather than bundled here.

Deliberately not doing:

• Rewriting the attach modules into one. The surface is ~2900 lines across three files plus a module and test file per mode; the failure modes above are all local.
• Client-side keystroke replay after a reconnect. Item 3 shows delayed replay is already the hazard; adding more of it would make a live agent execute stale input.
• Silencing any disconnect to make attach look solid. Every change above either keeps the message or makes it more specific.

[khaliqgant]

Item 7 root-caused: the fleet hint queries the wrong registry

Reproduced on this machine, attaching to an agent that lives on sf-mini without --node:

$ agent-relay node agent attach sandbox-lead-claude-0820b --mode view
Error: no agent named 'sandbox-lead-claude-0820b'     # stderr, 50 bytes
$ echo $?
1

Two findings.

The "near-empty stream" is a measurement artifact. stdout is 0 bytes and the CLI exits 1 with the error above on stderr. A byte count over merged streams reads as ~54 bytes of "stream"; there is no silent success. Worth stating plainly so nobody hunts a phantom.

The real defect is that the error does not name the node, even though fleet-hint.ts exists to do exactly that. The hint returns null, so attach falls back to the indistinguishable "no agent named X".

Why: resolveFleetHint looks the agent up with relay.agents.get(name) and reads metadata.fleet.nodeId. That is the workspace agent registry, and a broker-spawned worker on a fleet node is not in it:

$ agent-relay agent list | grep -c sandbox-lead-claude-0820b
0
$ agent-relay fleet agent list | grep -c '"node": "sf-mini"'
5

The placement data is right there in the fleet node inventory; the hint just asks the wrong source, gets a 404, and swallows it (by design — every failure path returns null). So the one case the hint was written for is the one case it cannot answer.

Fix shape: when agents.get returns nothing, fall back to the fleet inventory (the same source agent-relay fleet agent list uses) before giving up, and keep the existing "run agent-relay node agent attach <name> on that machine" wording. Small and self-contained; not bundled into #1598, which is scoped to the disconnect paths.

[khaliqgant]

Item 4 (events-WS reconnect): feasibility read and design

The infrastructure is better than I expected, but it constrains the design in one decisive way.

What already exists

• The broker annotates every durable broadcast event with a monotonic seq (replay_buffer.rs) and /ws accepts both ?sinceSeq= and ?since_seq= (listen_api.rs:372-380).
• Replay-on-connect is implemented, including an explicit replay_gap frame when the buffer cannot cover the requested range (catch_up_after_lag, handle_dashboard_ws). A resuming client is told when it missed something rather than silently losing it.
• The CLI already threads a cutoffSeq through attach and has a StreamSyncBuffer that reconciles replayed chunks against a snapshot offset.

The constraint: screen output is not replayable

broadcast_if_relevant classifies worker_stream and delivery_active as ephemeral — broadcast but never stored:

let is_ephemeral = matches!(kind, "worker_stream" | "delivery_active");

So a reconnect can backfill durable events (delivery counters) but cannot replay the screen bytes missed during the outage. That is fine, and arguably better: for a terminal the authoritative state is the grid, not the byte history. The resume should re-fetch the snapshot and repaint, which is what attach already does on first subscribe.

Design

On an unexpected close (any code other than 1000/1005), instead of finish(1):

1. Suspend input forwarding immediately (drive/passthrough). While the screen stream is down the operator would be typing blind into an agent they cannot see — and unlike the input-stream outage, the input socket is a separate socket that may still be perfectly alive. This is the one genuinely new safety rule in the design.
2. Show a reconnecting state in the status line rather than a bare red error line.
3. Reopen /ws?sinceSeq=<highest seq seen> with bounded exponential backoff, mirroring attach-input-recovery.ts (attempts, backoff cap, per-attempt deadline).
4. Verify worker identity before resuming, fail closed, exactly as the input stream does. A name is not an identity; if the worker was replaced, a "successful" resume would show one agent's screen while keystrokes went to another.
5. Re-fetch the snapshot, repaint, re-seed predictive echo, re-establish the scroll region and status line, reconcile buffered chunks by offset.
6. Re-assert resize ownership and this session's inbound delivery mode (the broker's idle-takeover may have reclaimed both).
7. Resume input forwarding. On exhaustion, exit non-zero with today's message — a real outage must still end the session loudly.

Honest scoping note. onSubscribed is re-runnable in outline but interleaves one-time attach steps (opening the input stream, taking stdin, entering raw mode) with per-subscribe steps, and the session loop currently assumes a single socket for the lifetime of settled / finish. Splitting those two concerns is the bulk of the work, and it has to be done once per mode (drive, view, passthrough) since each owns its own socket wiring. This is a real refactor, not a patch — which is why it is not in #1598.

(Edited: an earlier revision of this comment was mangled by unescaped backticks in my shell command, which pasted terminal output into the body. Content above is the intended text.)

[khaliqgant]

Two corrections to this map, after further reading

1. Item 3 ("stale input replay") is mostly legitimate type-ahead — I overstated it

The observation stands: writes already accepted into the broker's drainer queue are not discarded, and they reach the child whenever it resumes reading, which can be minutes later. The SIGSTOP → type → SIGCONT → dump-pty reproduction is real.

But calling that "stale replay" without qualification was wrong, and it would lead to a bad fix. Typing ahead into a busy agent is a normal, desirable workflow — the operator types their next instruction while the agent finishes a tool call, and expects it to land when the agent is ready. That is precisely what the queued write does. A TTL that discarded interactive input after N seconds would break the common case to protect the rare one.

The genuinely hazardous case is narrower: input the operator has been told was lost, or believes was lost, and therefore retypes — so both copies land. Which means the fix is not a TTL, it is never claiming input was dropped unless it provably was:

• A refused write (pty_write_queue_full) is provably never enqueued, so reporting it as dropped and rolling the echo back is correct — that is what fix(attach): stop tearing down the input stream when an agent is busy #1598 does.
• A worker_timeout write may still land, so Drive input stream repeatedly drops and reconnects on worker_timeout while the worker is merely busy #1544's rule of not rolling the echo back is also correct.
• An outage on the input socket sits between the two, and today the CLI drops those keystrokes client-side and says so. That is defensible, and it is the one place a TTL discussion could reasonably start — but it needs evidence about real operator behaviour, not a guess.

I am not proposing a change here. I am recording that the earlier framing pointed at a fix that would have made attach worse.

2. Item 4's blast radius is smaller than I implied on the --node path

I wrote that the missing events-WS reconnect was my best explanation for the unexplained mid-session disconnects. Having now read the fleet loopback shim (attach-fleet-node.ts), that is only partly true, and not at all in the deployment where it was reported:

• On a --node attach the client's events WS terminates at the local loopback shim, not at the remote broker. When the shim's upstream connection to Relaycast drops, it reconnects with backoff (6 attempts, ~31 s budget) and does not close the client's event sockets while doing so. A transient cloud-leg blip therefore does not end the session.
• The client's session only dies once the shim gives up, and that path closes with code 1011 and a detailed reason naming the node, endpoint, attempt count and backoff budget.

So the client-side non-reconnect matters for: a local broker restart, a shim that has exhausted its retries, and any close the shim did not originate. It is still a real gap — one 1006 ends a session that could be resumed, and the resume machinery (?sinceSeq=, replay_gap, StreamSyncBuffer) already exists — but it is no longer obviously the cause of what was reported.

What would settle it: the exact [drive] connection closed (code: …, reason: …) line from a real disconnect, plus whether the attach used --node. A 1011 with a reason means the shim exhausted its retries and the fix is in the shim's budget; a bare 1006 means the socket died under us and the fix is the client-side resume. Building the resume refactor before knowing which would be guessing, and this investigation has already cost one wrong lead chased from a symptom string.

[khaliqgant]

Item 8 (agents older than ~5h): attach does NOT share route or session state with DM delivery

The specific question was whether attach shares anything with the DM-delivery path, given relay#1593 reports delivery failing to agents past roughly five hours. Answering from the code, because it is unambiguous there:

They are deliberately separate lanes. crates/broker/src/terminal_control.rs opens its own Relaycast connection, and says why in its module doc:

Dedicated Relaycast terminal transport. This deliberately does not share node_control: terminal output can be continuous and subject to backpressure, whereas node registration, heartbeats, and action delivery need a small independent control lane.

DM delivery rides node_control / /v1/node/ws. Attach rides the terminal lane. Different sockets, different reconnect state, different session bookkeeping.

The terminal lane is also independently defended against the exact failure mode an age-correlated bug would suggest: a 12 s ping with a 48 s read-idle cutoff, added precisely because a hibernatable Cloudflare Durable Object (or any proxy) can drop an idle socket without ever sending a close frame, leaving a lane that looks connected but is dead.

Where the two paths do touch, and the only places worth suspecting:

1. Both ultimately write into the same per-worker PTY drainer queue (relay-pty), so a delivery problem that parks injections can consume slots an operator's keystroke later needs. That is items 1 and 2 of this map, not an age effect.
2. Both read and write the worker's inbound delivery mode, which an attach session asserts on connect and restores on detach.

Conclusion: an age-correlated DM-delivery defect should not, by construction, make attach behave differently for an older agent. I could not reproduce an age effect on attach, and I am not going to assert one exists on the strength of a correlation observed elsewhere. If attach really does behave differently past ~5 h, the evidence to capture is the same one line as item 4: the exact [drive] connection closed (code: …, reason: …) text, plus the agent's age and whether the attach used --node.