readinessReconcile timeouts are ordered so the deadline can never fire: 90min budget, 10min stall, ~50min container life

[khaliqgant]

Splitting this out of #372, where it surfaced while confirming the wedge in production. It is a distinct defect from "the breaker cannot trip", and it is not addressed by #374.

The mismatch

Three timeouts govern a stalled readinessReconcile sweep, and they are ordered so that the system can never act on what it detects:

constant	value	source
DEFAULT_READINESS_RECONCILE_TIMEOUT_MS	90 min	schema.ts:62 (90 * 60_000)
READINESS_RECONCILE_STALL_INTERVALS → state: 'stalled'	10 min	public-health.ts:64/491
observed container lifetime	~50 min	measured, see below

The deadline that would convert a hang into an actionable failure is 90 minutes. The container is replaced at roughly 50 minutes. The deadline therefore never fires. Meanwhile the stall is declared at 10 minutes — so the system knows it is wedged for about 80 minutes before it would have been permitted to do anything, and is recycled first.

Evidence for the ~50 minute lifetime

Two independent surfaces agree:

1. Boot timestamps observed on /healthz across the incident.
2. cloud#3046 carries 51 dependency-park comments at ~32–68 minute spacing, mostly an identical dedupe signature. The dedupe map (#dependencyParkNotices, factory.ts:8941) is in-memory and is never cleared for a still-parked issue — so each repeated comment requires a fresh process. 51 comments ≈ 51 container lifetimes.

Why #374 does not cover this

#374 bounds the sweep, which is the right mechanism — but sweepBudgetMs defaults to reconcileTimeoutMs, i.e. the same 90 minutes, and the deployed template sets no override. So the default inherits exactly this mismatch and the bound never fires. #374 plus an explicit sweepBudgetMs near the stall threshold would fix the immediate outage; this issue is about the defaults being self-defeating rather than about any one call site.

Suggested shape of a fix

The relationship between "we declare it stalled" and "we are allowed to act" should be expressed rather than left to three independently-chosen constants. A default deadline that exceeds the process's own expected lifetime is never a real deadline, and is worth asserting against.

[khaliqgant]

Correcting my own framing in the body above: I suggested an explicit sweepBudgetMs "near the stall threshold" (10 min). I withdraw that figure.

schema.ts:76-82 documents that a budget below realistic cold-mirror hydration converts a slow boot into a loop that never makes progress — the sweep checkpoints only at the end, so each restart begins from zero — and #36 measured 61 minutes for that hydration in production. The budget covers startup backfill too. A 10-minute budget sits close to that cliff, and a boot loop is a worse failure than the stall it would fix.

~15 minutes ("liveSubscription": { "sweepBudgetMs": 900000 }) is the better balance: far above the observed warm sweep (lastDurationMs: 917 — under a second), below container lifetime so it can actually fire, and further from the #36 cliff.

Also correcting two supporting claims of mine: the ~50 minute container lifetime is not independently measured (one container observed alive at 34 min — treat as unknown; the argument only needs it well under 90 min), and container-env.mjs does not exist — the no-override finding stands on an org-wide search returning 0 hits for these keys in factory-cloud, plus /healthz reporting the schema-default intervalMs: 60000.