[factory] A hang is not a failure: readinessReconcile can never trip its own breaker, so a wedged sweep is a permanent absorbing state

[khaliqgant]

Filed from the 2026-08-25 dispatch outage. This is the reason it lasted days
rather than minutes, and it is independent of any individual unbounded call.

The defect

readinessReconcile has a failureThreshold of 3 and a consecutiveFailures
counter. Through every wedge observed today, across five container boots and
two different root causes, that counter read 0.

08:24:00Z   rr=stalled   inFlightMs=1860000 (31.0 min)   missedPasses=30
            consecutiveFailures=0            failureThreshold=3
            intervalMs=60000

A sweep that never returns never throws. A hang is not a failure. So the
threshold cannot be crossed, the breaker cannot open, and any recovery gated on
consecutive failures — including container replacement — can never fire. The
subsystem is stuck in a state its own health model has no way to escape.

The public projection even reports ok: true alongside
status: "degraded" while dispatch is fully down.

Why the existing signals did not help

• consecutiveFailures — structurally cannot move during a hang.
• state: stalled — it is set, but nothing acts on it.
• missedPasses — climbs correctly (one per interval) and is the honest
  signal, but nothing acts on it either.
• inFlightMs — climbs exactly with wall clock and is the sharpest indicator
  available. Also unconsumed.

So the instrument already carries the truth. Nothing consumes it.

What done looks like

1. A hang is classified as a failure. If inFlightMs exceeds some multiple
   of intervalMs (the observed wedges ran 30x and 31x over a 60s interval),
   the sweep is unhealthy in the sense the breaker understands — not merely
   labelled stalled.
2. Recovery is reachable. Whatever action consecutiveFailures >= failureThreshold unlocks must also be reachable from a hang. Today a wedged
   sweep is a permanent absorbing state; see also factory-cloud#62, which
   makes the same point about /healthz being unable to see an offline
   dependency, and factory-cloud#63 on recovery being gated on a
   timer-written heartbeat.
3. A must-fire test: a sweep whose inner call never returns eventually trips
   the same path a failing sweep trips. It must fail before the change.
4. A must-not-fire test: a sweep that is merely SLOW but completing inside
   its budget does NOT trip it. Without this, the obvious wrong fix — treating
   any long sweep as failed — passes and introduces flapping.

Scope note

This is deliberately NOT a fix for any particular unbounded call. Today's
outage went through three of those:

• factory-cloud#78 — unbounded FACTORY_STATE Durable Object calls
• factory#368 — unbounded relayfile change-log tail reads
• and a third layer, the retry of the now-bounded call

Each was real and each was fixed, and the wedge returned one layer down every
time. A detector that treats a hang as a failure is what makes layer N+1 a
degradation rather than an outage, regardless of which call is next.

[khaliqgant]

Confirmed in production — this is live right now, not theoretical

Measured today from /healthz (unauthenticated read; bogus-route control returns 404 as required). Four samples:

	s1	s2	s3	s4
inFlightMs	675,023	735,049	780,084	795,096
missedPasses	11	12	13	13
consecutiveFailures	0/3	0/3	0/3	0/3

• factoryProcess: running, heartbeat updating — the process is not dead.
• lastStartedAtMs is identical across all samples: one sweep, begun 11:48:55Z, never returned. inFlightMs tracks wall clock 1:1.
• discoveryDeferred: "sweep-in-flight", degradedSubsystems: [readinessReconcile].
• dispatchCapacity active=0 waiting=0 in 120/120 samples.
• Control: the same instrument reported healthy in 45 samples, so it is capable of reporting non-stalled.

The absorbing state is exactly as this issue describes: every later pass coalesces onto the hung sweep rather than around it (factory.ts:2982-2988 returns the in-flight promise), and because a hang throws nothing, consecutiveFailures stays pinned at 0 against failureThreshold: 3. The breaker cannot fire.

Observed impact: no issue has been dispatched for at least 39 hours. Two canaries filed to match the deployed safety gate exactly — #350 (~39h) and #364 (~20h) — have zero comments and no label changes. I ruled out the usual false explanation first: factory is in repos.names, and both satisfy requireLabel: "factory" and requireTitlePrefix: "[factory]". They pass the gate as written.

A restart does not fix it. The 11:44:39Z boot bought 4 sweeps / ~4 minutes before re-wedging at 11:48:55Z, and dispatched nothing in that window.

A related gap that I do not think is captured here or in #374

The one mechanism that would convert this hang into a failure is the #296 readiness deadline, and the numbers do not line up:

• DEFAULT_READINESS_RECONCILE_TIMEOUT_MS = 90 * 60_000 (schema.ts:62) — 90 minutes
• state: 'stalled' is declared at 10 minutes (READINESS_RECONCILE_STALL_INTERVALS = 10, public-health.ts:64/491)
• observed container lifetime — ~50 minutes

The container is replaced before the deadline can fire, so the system detects the stall roughly 80 minutes before it is permitted to act on it. That mismatch may warrant its own issue.

Nothing was restarted, redeployed, or merged to produce this.