#343 regressed start-disabled deploys: run-once --dry-run exceeds its 10-minute budget

[khaliqgant]

#343 (b7c68462, shipped in 0.1.70) regressed the start-disabled deploy path. Controlled A/B, same host, minutes apart, only the pinned version differing.

deploy run	factory	mode	outcome
32661804498	0.1.69	start-disabled ("phase": "ready")	success — runOnceDryRun completed
32660636060	0.1.70	start-disabled ("phase": "failed")	failure — runOnceDryRun exited null (SIGTERM), durationMs: 600034
32657123745, 32653222936	—	live ("phase": "running")	success — live deploys skip runOnceDryRun entirely

evidence.phase = 'ready' is set only after runFactoryProbe('runOnceDryRun', …) returns inside the FACTORY_START_ENABLED !== '1' branch (container/entrypoint.mjs:1049-1053), so phase: ready on 0.1.69 is positive proof the sweep ran and finished. That is the known-present-needle this comparison needed.

Mechanism

From the failing container's own stderr:

[factory] could not establish orphan-recovery safety context; preserving in-progress issues
{"error":"Refusing to register relay agent \"factory-cloud-1c88bf23\": this fleet client is read-only. ..."}

The read-only client correctly refuses to mint an identity — that part is #343 working as designed and it is what ended the dispatch outage. But #githubOrphanRecoveryContext() (src/orchestrator/factory.ts:2850) reads fleet.roster() regardless of dryRun, so on a GitHub-backed sweep it now fails, orphan recovery is skipped, and the sweep preserves in-progress issues instead of resolving them. The working set stays larger and the sweep exceeds its 10-minute budget.

Scope — read this before panicking

• Live dispatch is NOT affected. isReadOnlyFactoryCommand classifies a live run-once/loop/dispatch as not read-only, so those construct a registering client, orphan recovery works, and nothing changes. The two live deploys above confirm the live path is green.
• What IS broken is the pre-cutover validation gate. A start-disabled deploy is precisely how we prove a full cycle runs against the real host without side effects, and it now always fails. Losing that is worse than it sounds: it removes the safe way to validate a Factory build before going live.

Deliverable

A dry run should not need a workspace identity to decide what it would do. Preferred fix: make orphan-recovery context honour dryRun — skip it and report the sweep as degraded, rather than attempting a roster read that cannot succeed under a read-only client. Skipping work should make the sweep faster, so if it does not, measure where the 10 minutes actually goes before changing the budget.

Explicitly not acceptable: raising the 10-minute timeout to make this pass. That hides the regression rather than fixing it, and the budget was already tightened deliberately (entrypoint.mjs documents the 30→10 minute reasoning, including that a preflight stall is the outage it is meant to prevent).

Also worth a look while in there, both from the same failing sweep and probably independent:

• relayfile listTree on .integrations/discovery fails with missing required scope: fs:read
• a repeating local mount at .../.integrations is stale (last reconcile 1m ago); refreshing loop

Provenance

I merged #343 and I own this regression. My first two readings of it were wrong and are corrected here: I initially compared against nine live deploys that never ran this probe at all, then concluded from a grep for run-once started that no deploy had ever run the sweep — that grep only matches when the probe fails, because the deploy log dumps stderrText only on failure. The phase field is the correct discriminator.

[khaliqgant]

A dry_run=true deploy CANNOT succeed on a cold container — and this corrects #348, which I filed against myself

The arithmetic

• runOnceDryRun budget: 10 minutes — container/entrypoint.mjs:1051, { timeoutMs: 10 * 60_000 }
• Cold-mirror ingest: ~33 minutes — measured by factory-dispatch-lead-0823, and independently consistent with the lastDurationMs: 2201002 (36.7 min) sweep I observed

A freshly-booted container cannot finish that probe inside its budget. The failure is guaranteed, not intermittent:

"failure": { "code": "bootstrap_failed", "duringPhase": "preflight",
             "message": "runOnceDryRun exited null (SIGTERM)" }

And because runOnceDryRun sits inside if (process.env.FACTORY_START_ENABLED !== '1') (:1031), it runs only on start-disabled deploys. A live deploy skips it entirely.

Correction to factory#348

I filed #348 claiming #343 regressed the start-disabled deploy path, based on an A/B: 0.1.70 start-disabled failed, 0.1.69 start-disabled succeeded.

That comparison is unsound. The variable I controlled was the version; the variable that actually decides the outcome is whether the container's Relayfile mirror is cold or warm. My 0.1.69 "control" ran on a host that had been serving for a while; the 0.1.70 run was cold. I compared two versions and unknowingly compared two mirror states.

#349 (skip the orphan-recovery roster read on dry runs) was still a correct change and remains merged. But it did not and could not fix this, and the current failures on 0.1.73 — which contains it — prove that.

I am not closing #348, because a real question remains: is the 10-minute budget simply wrong for a cold boot, or should the dry run be warm-start-only? But its premise is wrong as filed, and anyone reading it should not treat #343 as the cause.

The operational consequence, live today

Two deploys today by kjgbot (12:33Z, 12:54Z) were start-disabled. Both failed on this probe, and both replaced a running container with one that never starts the daemon. Result: phase: failed, and zero factory-cloud-* identities online out of 31.

So dispatch was off for ~50 minutes, not because of any dispatch defect, but because start-disabled deploys are structurally unsurvivable on a cold container and each one evicts the working daemon.

I have dispatched a live deploy (32732274486) to restore it.

Recommendations

1. Do not run dry_run=true against production until the budget/cold-start mismatch is resolved. It cannot pass, and it takes dispatch down with it. This is a footgun aimed at exactly the person trying to validate safely.
2. Either raise the budget above the measured cold-ingest time, or gate the dry run on a warm mirror, or drop the probe from the deploy path and validate elsewhere. Do not simply raise it to 35 minutes without measuring — entrypoint.mjs already documents that a long preflight stall is the outage it exists to prevent.
3. The lane's lastReplacement: { atMs, reason } instrument would have made this visible immediately, by attributing each container replacement to the deploy that caused it.