[factory] Provision an on-demand sandbox node when no eligible node is live or all are saturated

[khaliqgant]

Problem

Factory's dispatch depends on a fleet node that already exists and is already live. There is no path that provisions compute on demand. Today that turns any single node failure into a total dispatch outage, and it turns node saturation into a queue rather than more throughput.

The placement surface is only two values:

src/fleet/internal-fleet-client.ts:119   readonly placementLocality = 'local'  as const
src/fleet/relay-fleet-client.ts:146      readonly placementLocality = 'remote' as const

local means this host. remote means some already-registered fleet node. Neither can bring capacity into existence, so when the rostered node is offline there is no third option — Factory waits.

The motivating evidence

factory-primary has been status=offline, handlers_live=0 with last_heartbeat_at frozen for hours, and Factory has dispatched nothing for days. The root cause is a genuine defect and is being worked separately.

But the outage is not only that defect — it is that the defect had nowhere to fail over to. One node stopped holding a WebSocket and the entire dispatch plane stopped. A system that could have provisioned an ephemeral node would have degraded to "slower and more expensive" rather than "stopped".

That is the argument for this issue, and it is independent of whatever fixes the WebSocket.

What is being asked for

Factory should be able to provision an on-demand sandbox node and dispatch onto it when:

1. no eligible node is live — the outage case; and
2. every eligible node is saturated — the throughput case, where work is queued behind max_agents while capacity could be bought.

Why this is smaller than it looks

This is plausibly routing logic, not a provider integration. Daytona already exists as a relay configuration concept and sandbox placement has been observed at the fleet layer — a fleet spawn has previously landed on a Daytona sandbox (it ignored worker_cwd and never received its task, which is its own bug, but the placement happened). And docs/sandbox-router-product-and-build-plan.md already sets the direction: one API across providers with cross-provider failover.

So the likely gap is that Factory has no decision to make that call — not that the call cannot be made. That should be verified before any design work, because it decides whether this is a week or a day.

Design constraints, learned the expensive way

• Fail closed on the claim, not on the capacity. A dispatch that cannot record its claim must still abort. Adding a capacity tier must not add a path that dispatches without a claim.
• An ephemeral node must not become a second dispatcher. The single-dispatcher invariant is what stops duplicate work; a provisioned node is a worker host, not another Factory.
• spawned: true is not running. A previous Daytona placement reported success while ignoring worker_cwd and never injecting the task, and the agent died in 75s. Any on-demand path needs an arrival proof — the agent reporting in — not a provisioning acknowledgement.
• Cost must be bounded and attributable. On-demand compute that fails open is a bill with no ceiling. State who pays, what the cap is, and what happens at the cap.
• Teardown must be guaranteed, including on the crash path. The failure mode that matters is the one where Factory dies holding a provisioned sandbox.

Definition of done

A dispatch that finds no live eligible node provisions a sandbox node, dispatches onto it, proves the agent arrived, completes the work, and tears the node down — with the claim recorded before the spawn and the cost bounded. Plus the saturation case: work that would queue behind max_agents provisions instead, under the same cap.

Not in scope

The current /v1/node/ws defect. This issue exists because that defect had no fallback, not to fix it.

[khaliqgant]

Correction from Khaliq: a sandbox node is no different from any other node

I filed this with the wrong framing and it makes the work look bigger than it is. Correcting on the record rather than editing quietly, because the correction is the useful part.

What I implied: that Factory needs a new placement tier — a sandbox concept sitting beside local and remote, with provider routing, capability negotiation, and a new dispatch path.

What is actually true: a sandbox node is just a node. It enrolls, registers, heartbeats, advertises handlers and accepts work through exactly the same fleet interfaces as sf-mini or factory-primary. Factory's existing placementLocality = 'remote' already means any registered fleet node — it does not care how that node came to exist.

What that changes

Nearly all of it. Factory does not need to learn about sandboxes, providers, or a third placement mode. The only real gap is that nothing provisions a node when capacity is short. Once something does, the new node joins the fleet and Factory's existing remote placement finds it with no dispatch changes at all.

So the requirement reduces to:

1. A trigger — Factory (or a supervisor) detects "no eligible node is live" or "all eligible nodes are saturated" and asks for one more.
2. A provisioner — something starts a machine and runs the node agent on it, so it enrols and registers like any other node.
3. Teardown — it goes away when the work does.

Steps 1 and 3 are the actual engineering. Step 2 is largely the node agent doing what it already does, somewhere new.

What survives from the original filing

The motivation is unchanged and, if anything, stronger: one node stopped holding a WebSocket and the entire dispatch plane stopped, because there was no way to bring capacity into existence. That argument does not depend on sandboxes being special — it depends on there being exactly one node and no way to make another.

The constraints also survive, and two get simpler under the correct framing:

• Claim-before-spawn still fails closed. Unchanged.
• A provisioned node is a worker host, not a second dispatcher. Now nearly free: it is just a node, and nodes are not dispatchers.
• spawned: true is not running. Still required — a previous provisioned placement reported success while never injecting its task. But the arrival proof is now the ordinary one: the node registers and advertises handlers, the same signal every other node gives. No bespoke check.
• Bounded cost and guaranteed teardown. Unchanged, and still the part most likely to be got wrong.

Revised definition of done

When no eligible node is live or all are saturated, a node is provisioned, enrols normally, takes the dispatch through the ordinary remote path, and is torn down afterwards — with the claim recorded before the spawn and cost bounded. No new placement mode in Factory.

If the trigger and teardown already exist somewhere in the fleet layer, this may be close to configuration rather than construction. That is worth establishing first, before anyone designs anything.