Fleet node control-plane connection blackholes: node alive and spawning, reports offline with activeAgents 0, placement fails

[khaliqgant]

Summary

A fleet node's control-plane connection blackholes silently: the node process stays alive, keeps its TCP sockets ESTABLISHED, continues spawning workers successfully — and stops heartbeating. The control plane then reports it offline / handlersLive: false / activeAgents: 0 while ps shows it running N agents, and every targeted spawn to that node fails with placement TTL expired.

This is not a dead node. It is a node the scheduler can no longer see.

Measurement, finn-mini, 2026-08-20

signal	value
node process	alive, pid 21570, uptime 22h30m, launchd com.agentrelay.fleet-node
ESTABLISHED sockets to Cloudflare	2 (104.18.13.48:443, 104.18.12.48:443)
workers actually running	5, all healthy, all doing real work
last line written to fleet-node.stderr.log	19:28
control-plane lastHeartbeatAt	19:29:33Z
control-plane status / live / handlersLive / activeAgents	offline / false / false / 0
time observed in this state	2h21m and counting

The node stopped writing to its own log and stopped heartbeating at the same moment, while its sockets stayed open. No error is logged at the transition — the last line is an ordinary success:

[agent-relay] spawned worker 'relay-1579-rebase-0820b' via relaycast

That spawn succeeded — the worker is alive and has since rebased a PR and answered review threads. So the node completed the operation and then went silent.

Impact

• New work cannot be placed on the node. Two targeted fleet spawn --node finn-mini attempts returned Placement queued: target node "finn-mini" is offline; placement TTL expired. Placement had to be moved to another host, including creating duplicate git worktrees there.
• Existing workers keep running, so the loss is invisible unless you ssh in. The fleet looks like it has capacity it cannot use.
• fleet agent list reports the node's agents as count only / names unavailable, because the node can no longer publish its roster — compounding relay#1585.
• Capacity concentrates on the remaining nodes with no signal that it has.

Why the existing signals do not catch it

• An ESTABLISHED socket is not a live session. Both sockets are open; nothing is flowing.
• Worker spawn success is not control-plane health. The node kept spawning right up to the silence.
• activeAgents: 0 is confidently wrong — there are five. So the field a scheduler would trust is not merely stale, it is inverted.
• Nothing is logged. A grep of the node's stderr for provider websocket|connection closed|failed registering|reconnect|heartbeat returns only entries from a previous incident hours earlier, nothing at the transition.

Asks

1. Detect it. The node should notice its own control-plane link has gone quiet — an application-level ping with a deadline, not TCP liveness — and reconnect. relay#1462 was merged to detect this class; this is either a gap in it or a regression against it.
2. Log the transition. A node that stops heartbeating must say so in its own log. Today the only way to discover this is to compare lastHeartbeatAt against ps on the host.
3. Do not report activeAgents: 0 for an unreachable node. Zero is a claim. unknown is the truth, and the distinction decides whether a scheduler routes work there.
4. Consider failing the placement fast and loudly. placement TTL expired after a wait reads as congestion; node unreachable would have pointed straight here.

Environment

• broker 11.8.0, node launchd-managed (com.agentrelay.fleet-node), --state-dir ~/.agentworkforce/relay/finn-mini-node/state
• Same host previously logged failed registering agent with RELAY_WORKSPACE_KEY workspace key → Invalid response: Max retries exceeded during the D1 saturation window, so a registration/handshake path is already known to be fragile here.

[miyaontherelay]

Claimed by relay-reconnect-fix-0825.

Audit complete against origin/main at 191e5f1, merged PR #1462, current node-control/reconnect code, all open PRs, and unmerged/relevant remote branches.

Findings:

• fix(broker): detect a blackholed node-control connection instead of trusting writes #1462 is present in production-era main, but its deadline refreshes on any inbound WS frame. A peer that keeps polling the socket automatically answers pings, so transport pong traffic can mask the measured application failure where node.heartbeat is no longer acknowledged.
• Node-control connect_async and Sink::send awaits are unbounded; either can prevent the watchdog select loop from observing its deadline.
• Reconnect backoff is capped at 30s, but successful sessions do not reset it and transition logging is incomplete.
• Relay SDK normalization currently preserves engine history activeAgents: 0 even when status=offline/live=false, presenting an unknown count as a fact.
• No current open PR or unmerged remote branch contains an overlapping node-control/application-heartbeat fix.

Plan:

1. Add a deterministic regression that keeps the process/task alive and TCP/WS open and polling (pongs continue), but withholds application heartbeat replies; require the client to declare the link dead and reconnect.
2. Correlate heartbeat request IDs with engine replies and enforce an application-level acknowledgement deadline independent of WS ping/pong.
3. Bound connect and write awaits, keep capped reconnect backoff, reset it after a real session, and log explicit connected -> unhealthy/disconnected -> reconnect-scheduled transitions with reasons/delay.
4. Normalize unreachable nodes active-agent count to unknown on the Relay SDK surface.
5. Run focused broker + SDK tests, then full relevant package checks, push a focused branch, and open a PR.

[miyaontherelay]

Pre-fix must-fail regression recorded on untouched node-control logic at origin/main 191e5f1.

Command:
cargo test -p agent-relay-broker --lib node_control_disconnects_when_application_acks_stop_but_socket_stays_live -- --nocapture

Result: FAILED (exit 101), 0 passed / 1 failed / 1029 filtered out, 2.01s.

Failure:
application-level liveness deadline did not fire: Elapsed(())

The fixture kept the client task/process alive with active_agents=3, held TCP/WS open, continuously polled the socket so automatic pong traffic continued, drained node.heartbeat and inventory.sync frames, and withheld all application replies. Current #1462 logic stayed connected because pong frames refreshed last_inbound.

[miyaontherelay]

Implementation is published in #1610 at exact head 44ddb17dcafcdefa347cb31bb132a3dc4e9aa846.

The fix keeps the #1462 WebSocket read-idle watchdog and adds a distinct application deadline based on correlated replies to the existing inventory.sync request (the current engine does not reply to node.heartbeat). In production the application deadline is 120 seconds. Connects and writes are bounded at 15 and 10 seconds, reconnect delay remains capped at 30 seconds, and logs now identify transport-connected, application-ready, unhealthy, and reconnect-scheduled transitions. The SDK also omits activeAgents when the node is offline/unreachable while preserving a real zero for reachable nodes.

Verification at this head:

• deterministic live-socket/dead-application test: passed
• acknowledged positive control: passed
• node-control filter: 11 passed, 0 failed
• full broker library: 1027 passed, 0 failed, 4 ignored (with worker-injected attestation/Git-hook variables removed)
• Rust fmt and clippy with warnings denied: passed
• focused SDK normalization: 2 passed, 0 failed
• affected messaging normalization: 1 passed, 21 skipped
• touched SDK Prettier check: passed

Local full-SDK caveat: an isolated npm ci hung before creating node_modules; the shared checkout dependency tree produced 25 unrelated failures from mixed Zod copies and worker-injected origin metadata, while both changed SDK paths passed. PR CI is running against a clean environment.

Cross-check: the explicit registration state machine in AgentWorkforce/relaycast#301 remains separate, and no incompatible wire change is introduced. AgentWorkforce/relaycast-cloud#85 did not overlap this broker-side recovery path.

No node restart, agent release, merge, deployment, or credential operation was performed.