HarnessDriverClient.connect() resolves the broker URL from connection.json once
and hands it to a transport that holds it as private readonly baseUrl for the life
of the client. Every consumer of this library inherits that, so a broker rebind
wedges all of them.
packages/harness-driver/src/client.ts:321-362 — readFileSynces
connection.json once, validates it, passes conn.url into a new client as
baseUrl. Static factory, nothing re-entrant, nothing re-reads the file later.packages/harness-driver/src/transport.ts:412,428 — stores it as
private readonly baseUrl. HTTP calls (:459), the events WebSocket URL
(:440) and the PTY input stream URL (:444) all derive from that one immutable
field.packages/harness-driver/src/transport.ts:502-527 — the WS auto-reconnect
reconnects to the same dead URL, so "it reconnects" does not help.
This is the root of the whole class. Six downstream consumers were found caching
because of it (factory, chief-app's stream bridge, skip, relayscribe, the
watchdog-recorder sidecars, oh-my-pi). Fixing it here would fix all of them at once;
each of those has a sibling issue filed as a fallback.
Background
The relay broker binds an ephemeral port (AGENT_RELAY_BROKER_PORT=0), so it
picks a new port every time it restarts and rewrites
.agentworkforce/relay/connection.json. Any long-lived process that resolves the
broker URL once and holds it therefore calls a dead port for the rest of its life.
This was observed in production in Factory: a daemon started at 22:33 kept using
the port from boot after the node restarted at 22:45. 1469 of 3000 log lines were
fetch failed, and it never self-recovered. Node restarts are routine, so caching
an ephemeral port for a process lifetime is broken by design.
Filed from the blast-radius survey on AgentWorkforce/factory#291, which swept every
connection.json consumer across the org. This repo was classified cached. I
re-read the file above against current source before filing; it matches the survey.
Suggested fix
Re-resolve on failure rather than capturing once. The shape that works, and the one
Factory shipped in AgentWorkforce/factory#299:
- On a broker call failure, re-read
connection.json and compare it to the broker
you are attached to (url + api key + pid).
- Same broker → it is simply down. Propagate the error. Do not reconnect —
this is what stops repeated failures becoming a reconnect loop.
- Different broker → it rebound. Drop the dead transport, reconnect from the
file, and re-establish any event subscriptions.
- Retry the call only if it is a read. Never replay a write the vanished broker may
already have accepted.
- Include the attempted base URL in the error. Node's
fetch failed is a bare
TypeError with the address buried in cause, which is exactly why the original
incident took a night to diagnose.
Cost is nil on the healthy path: nothing is read while calls succeed. Measured, the
failure-path read + parse of the 121-byte file is 66 µs.
There is also a working reference implementation in chief-app:
node/src/main.ts:162-186 (reconnect on a failed listAgents()) and
node/src/node.ts:106-130 (createLiveRoster).
HarnessDriverClient.connect()resolves the broker URL fromconnection.jsononceand hands it to a transport that holds it as
private readonly baseUrlfor the lifeof the client. Every consumer of this library inherits that, so a broker rebind
wedges all of them.
packages/harness-driver/src/client.ts:321-362—readFileSyncesconnection.jsononce, validates it, passesconn.urlinto a new client asbaseUrl. Static factory, nothing re-entrant, nothing re-reads the file later.packages/harness-driver/src/transport.ts:412,428— stores it asprivate readonly baseUrl. HTTP calls (:459), the events WebSocket URL(
:440) and the PTY input stream URL (:444) all derive from that one immutablefield.
packages/harness-driver/src/transport.ts:502-527— the WS auto-reconnectreconnects to the same dead URL, so "it reconnects" does not help.
This is the root of the whole class. Six downstream consumers were found caching
because of it (factory, chief-app's stream bridge, skip, relayscribe, the
watchdog-recorder sidecars, oh-my-pi). Fixing it here would fix all of them at once;
each of those has a sibling issue filed as a fallback.
Background
The relay broker binds an ephemeral port (
AGENT_RELAY_BROKER_PORT=0), so itpicks a new port every time it restarts and rewrites
.agentworkforce/relay/connection.json. Any long-lived process that resolves thebroker URL once and holds it therefore calls a dead port for the rest of its life.
This was observed in production in Factory: a daemon started at 22:33 kept using
the port from boot after the node restarted at 22:45. 1469 of 3000 log lines were
fetch failed, and it never self-recovered. Node restarts are routine, so cachingan ephemeral port for a process lifetime is broken by design.
Filed from the blast-radius survey on AgentWorkforce/factory#291, which swept every
connection.jsonconsumer across the org. This repo was classified cached. Ire-read the file above against current source before filing; it matches the survey.
Suggested fix
Re-resolve on failure rather than capturing once. The shape that works, and the one
Factory shipped in AgentWorkforce/factory#299:
connection.jsonand compare it to the brokeryou are attached to (url + api key + pid).
this is what stops repeated failures becoming a reconnect loop.
file, and re-establish any event subscriptions.
already have accepted.
fetch failedis a bareTypeErrorwith the address buried incause, which is exactly why the originalincident took a night to diagnose.
Cost is nil on the healthy path: nothing is read while calls succeed. Measured, the
failure-path read + parse of the 121-byte file is 66 µs.
There is also a working reference implementation in
chief-app:node/src/main.ts:162-186(reconnect on a failedlistAgents()) andnode/src/node.ts:106-130(createLiveRoster).