Agents spawned on remote fleet nodes die young and produce nothing, while spawn reports success

[khaliqgant]

Agents placed on remote fleet nodes register, then die young, producing nothing — and spawned: true hides it

Two independent instances today, on two different nodes, both confirmed dead by attaching rather than by reading a status field.

Instance 1 — sf-mini

spawn factory-placement-why-0823  target_node sf-mini
  -> {"spawned": true}                         (and a duplicate invocation refused: spawn_agent_name_in_use)
  -> registers, roster node_id node_203549044126121984   (correct node)
  -> lastSeenAt 2026-08-23T10:20:14Z
  -> attach at 10:32Z:
     "No active agent named 'factory-placement-why-0823' on node 'sf-mini'"
     HTTP 404, code agent_not_found

Output produced: none. No Step 0 report, no DM, no issue comment, no commit. It owned the critical path of a live outage for ~20 minutes and evaporated.

Instance 2 — daytona-1538-verify-0817b

spawn factory-322-guard-sandbox-0823  target_node daytona-1538-verify-0817b
  -> {"spawned": true, "agent_id": ...}
  -> registers, roster node_id node_214899219332034560   (correct node)

Output produced: none. Its target PR is byte-identical to before it was spawned.

Controls

• Both were verified remote, not silently placed locally: roster node_id matched the requested node in each case, and a local process search found zero harnesses for either name (the same search returns hits for local lanes, so the instrument works).
• Local lanes spawned the same way, in the same session, with comparable briefs, survive and produce work — five of them merged PRs today. So this is specific to remote placement, not to the briefs or the workload.
• Both briefs opened with a mandatory Step 0 environment report explicitly designed to fail loudly ("if you cannot clone or auth, say so and STOP — that is a valuable result, not a workaround"). Neither arrived, so this is not a lane deciding to stay quiet.

Why this matters more than it looks

1. spawned: true is not arrival. It reports that placement was accepted, not that an agent is alive and has its task. Every consumer of that field currently treats it as success.
2. The failure is silent in both directions. The agent produces nothing, and the registry keeps a plausible-looking record — lastSeenAt was recent and status read offline, which is the same thing the registry says about agents that are demonstrably alive. Nothing distinguishes "died young" from "thinking".
3. It defeats distribution entirely. Concentrating work on one node is a single point of failure — but distributing it currently means the work silently does not happen. Today that cost the critical path of a 5-day outage roughly 20 minutes and a stalled PR.
4. It blocks on-demand node provisioning (see [factory] Provision an on-demand sandbox node when no eligible node is live or all are saturated factory#336). Provisioning a sandbox node is not the hard part; arrival proof is. A node that comes up and accepts placements whose agents then die is worse than no node, because placement reports success.

What would close this

• An arrival proof distinct from placement acceptance: the agent reporting in under its own identity. Until an agent speaks, a spawn is pending, not successful.
• A death signal. Right now the only way to distinguish a dead remote agent from a working one is to attach and read the error. That should not require a human.
• Ideally, the reason. These may be dying for a mundane cause — no credential, no disk, no model credits, a missing runtime — but nothing surfaces it, which is the actual defect. A remote lane that cannot start should say why.

Reproduction

Spawn any agent with target_node set to a live remote node and a brief whose first instruction is to send a DM. Observe spawned: true, observe registration, then attach after ~15 minutes. Compare against the identical spawn with target_node set to the local broker, which survives.

[khaliqgant]

Existing readiness machinery is not wired to raw MCP spawns

Read-only source audit found a narrower arrival-proof gap that can be closed using code already in the repo.

• packages/cli/src/cli/agent-relay-mcp.ts:1222-1225 waits through invokeVerifiedPersonaSpawn(...) only for persona spawns. A raw CLI spawn returns the immediate action ack. The tool description explicitly calls this "asynchronous acknowledgement behavior."
• crates/broker/src/runtime/fleet.rs:1291-1320 already supports a stronger contract: when harness metadata contains verify_ready, it keeps the invocation pending until worker_ready, and maintenance fails it on early exit/readiness timeout.
• packages/cli/src/cli/agent-relay-mcp.ts:164-202 already has the bounded invocation poller used for persona spawns.

So placement acceptance and arrival proof are already separate internally, but the default raw spawn tool chooses the weaker path. Wire raw spawns through the broker verify_ready mode and the same bounded poller, and make the returned shape say ready: true only after the broker observed worker readiness. Must-not-fire controls: process created but no worker_ready; early exit; registration-only identity with no harness readiness.

This closes "spawned: true is not arrival," but not the whole reported failure: both observed workers may have become ready and then died young. The second half needs a correlated post-readiness exit signal carrying invocation/agent/node plus exit code/signal/reason, rather than collapsing that evidence into generic offline status. Broker maintenance already observes exits; the missing piece is preserving and exposing that reason after the initial invocation completes.

[khaliqgant]

The bounded arrival-proof fix is now PR #1606 at exact head 015a9adb503d4fe4af4ed6670f686c6a42f3704c.

It wires raw MCP CLI spawns through the broker's existing verified-readiness state machine:

• the MCP request sets a top-level verify_ready: true action flag;
• the broker accepts that flag without requiring a synthetic harness config;
• the MCP call polls the invocation to a terminal result;
• a legacy or unsupported node returning only { spawned: true } is rejected because it lacks ready: true;
• early-exit failures such as spawn_harness_not_ready are surfaced to the caller.

Local evidence: 54/54 relevant CLI tests, full typecheck, cargo fmt --check, and the complete broker package suite (1,040 passed / 4 ignored after removing only inherited agent-harness git/attestation variables that contaminate hook fixtures).

This does not close #1603. It fixes "placement acknowledgement is presented as arrival"; the durable correlated signal for a worker that becomes ready and then dies remains open.

I did not revive #1342: it is 451 commits behind current main, conflicts, and bundles readiness with dead-letter and strict-identity changes. Main already contains the newer per-generation readiness state machine, so #1606 is a one-commit adapter to that current mechanism rather than an obsolete broker rewrite.

[khaliqgant]

Correction to the preceding implementation update: PR #1606 exact head is 015a9ad. The earlier comment expanded the abbreviated SHA incorrectly; the PR URL, implementation, and test evidence are unchanged.

[khaliqgant]

Post-readiness exit path: the reason exists transiently, then is structurally lost

Read-only source audit at current relay checkout (2026-08-24T02:06Z) narrows the remaining half of this issue to one maintenance boundary:

• WorkerRegistry::reap_exited already returns (name, generation, code, signal, exit_reason). It detects wrapper exit, kill(0) disappearance, harness-pid death, and never-ready timeout before removing the WorkerHandle.
• runtime/maintenance.rs consumes that tuple and emits a transient local SDK agent_exited event with name/generation/code/signal/reason. This is the last point where the broker still has the correlated evidence.
• Relaycast receives only mark_agent_offline(name); the reason/generation never reach the durable central surface. Broker state then removes the agent.
• CrashInsights is not the answer as implemented: its record has only agent name/code/signal, writes uptime_secs: 0, omits generation/workspace/node/readiness/spawn invocation/reason, and crash-insights.json is saved only during orderly broker shutdown. A remote node/process death can therefore erase the very record needed to explain it.
• GET /api/crash-insights is node-local and exposes that incomplete in-memory store; it cannot answer “the exact fleet spawn became ready on node N, then died for reason R.”

Smallest durable closure

At the existing maintenance choke point, build one structured bounded WorkerExitRecord before removing state:

workspace_id, fleet_node_name, agent_name, generation, spawn_invocation_id, became_ready, spawned_at/ready_at/exited_at, exit_code, signal, exit_reason.

1. Carry spawn_invocation_id into WorkerHandle (or an equivalent generation-keyed correlation record) when the fleet action creates the worker; readiness currently removes the only pending invocation mapping.
2. Append the exit record atomically and save it immediately, not only at broker shutdown. A bounded 500-record file can reuse the crash-insights retention shape, but must not reuse its lossy schema.
3. Publish the same record through the existing hosted-agent event publisher as agent_exited in the worker workspace, so the central Relaycast event surface preserves it; the local API is a fallback, not the only evidence.
4. Preserve same-name generations independently. Never let a later respawn overwrite the prior generation’s death record.

Must-not-fire / durability controls:

• ready → nonzero exit: one record with exact generation/node/invocation/reason;
• ready → signal death and harness-pid death: signal or synthesized harness_exited survives;
• record → broker crash/restart: record reloads (proves per-exit persistence);
• same name respawn: both generations remain queryable;
• early verified-spawn failure: action failure and exit record correlate once, with no duplicate lifecycle publication;
• explicit release/clean task exit is classified distinctly from a crash.

This is narrower than a new supervisor: the broker already observes the truth. The missing work is retaining and exporting the evidence at the one point where it is complete.