fix(orchestrator): bound the whole sweep, not one more call inside it

[khaliqgant]

Do not merge — factory-lead holds the gate.

Three unbounded calls have wedged this sweep in a single day, on three different
transports. Each was real, each was bounded, and each time the wedge came back
one layer down. This PR does not bound a fourth. It makes the class survivable.

---

A. The aggregate budget

The property: no sweep can be in flight for longer than its budget, whatever
it is waiting on. Elapsed time is charged against one timer for the whole
pass, so it does not matter which await is slow, how many there are, or how
many times the sweep retries one of them.

Why a per-call bound cannot do this

bound	what it covers	what it misses
relayfileOperationTimeoutMs (#351/#368)	one relayfile call	the retry loop around it (L3); any other transport
reconcileTimeoutMs (#296)	the caller's wait, from outside runOnce()	the sweep itself keeps running, so every later cycle coalesces onto the wedged promise (runOnce(), the #runOnceInFlight branch, main factory.ts:2986-2989)
sweepBudgetMs (this PR)	the whole pass, from inside #runOnceWithDiscoveryFence	see does not cover below

Expiring from inside the fence is the entire difference. The sweep unwinds,
the discovery lease goes back, #runOnceInFlight clears, and the next cycle
claims a fresh lease and runs clean.

Mechanism, plainly — what it can and cannot interrupt

budget.run() is a race, not a cancellation. Same honest limitation #368
documented for withRelayfileCallDeadline, stated for the same reason.

• CAN abandon an in-flight await, on any transport, and unwind the sweep.
• CANNOT stop the abandoned work. The socket stays open, the SDK's own retry
  loop keeps running, and a side effect already in flight still lands.
• PARTIAL — budget.signal aborts at expiry, so anything that honours an
  AbortSignal is really cancelled. Nothing in the sweep consumes it yet: the
  relayfile client mints its own per-call signal and that file is owned by
  factory-wedge-layer2-0825 this week. It is exported so wiring it is one line,
  not a redesign.
• assertNotExpired() is a between-await check and is worth nothing against
  a call that never returns. What it buys is that an already-abandoned pass
  unwinds at its next loop iteration if it ever regains control, instead of
  running to completion beside the sweep that replaced it.

Teardown is bounded separately, on purpose

On the path that matters the budget is spent by construction, so teardown cannot
run under it — every step would reject and the lease would never be released,
and releasing it is the half that makes the next cycle clean. An unbounded
release would re-create this wedge one layer down, which is the pattern this
PR exists to end. It gets its own 30 s deadline. An abandoned release costs an
orphaned lease for one expiry window, which a later sweep reclaims as an orphan
(claim.reclaimedLease).

The default deliberately changes no timing

sweepBudgetMs defaults to reconcileTimeoutMs (90 min) and is clamped to it,
so no sweep that survives today is killed by this. The number is a policy
dial; the mechanism is the deliverable. Tightening it has a real cost: the
checkpoint commits only at the end of the pass, so a budget below realistic
cold-mirror hydration (#36 measured 61 min in production) turns a slow boot into
a loop that never makes progress — the trap reconcileTimeoutMs already
documents. Recovery inside 90 minutes therefore needs either a tighter
sweepBudgetMs at deploy time (your call, one config key) or the L3 retry bound
the other lane owns.

---

B. Is discovery hostage to the sweep?

The coupling you named is necessary, and it is not what cost us dispatch. A
different one is, and this PR removes it.

discoveryDeferred: 'sweep-in-flight' (main factory.ts:3071) — necessary, keep it

It fires only when claimDiscoverySweep finds another owner holding the
durable lease. What the lease protects is the discovery checkpoint: two
concurrent passes would both advance the same cursor via
#finalizeDiscoveryCheckpoint / completeDiscoverySweep, so one would commit a
watermark covering trees the other listed and the uncovered trees would never be
re-read. That is a correctness invariant, not a convenience.

And it is cheap. A deferred pass returns immediately (main :3062-3072) — it
does not block, it settles successfully, and it costs one interval of freshness.

Note it never fired during this incident: within one process runOnce()
coalesces rather than defers (main :2986-2989). Which answers your other
question —

Why discoveryDeferred went "sweep-in-flight" → None between 0.1.74 and 0.1.75

discoveryDeferred is a latched marker on the last SETTLED pass. It is
written only by #recordReadinessSweepOutcome, which runs only on a pass that
succeeded, and cleared only on the failure path (main factory.ts:2310).

• On 0.1.74 nothing settled during the wedge — the per-call deadline did not
  exist and reconcileTimeoutMs is 90 min. So an early boot-time deferral
  (previous incarnation's lease still live inside its 5-minute window) latched
  and froze on the surface for the whole outage.
• On 0.1.75 fix(mount): bound the relayfile change feed so a hung tail read cannot wedge the readiness sweep #368's deadline fires, so passes do settle — as failures — and
  :2310 clears the marker every time.

So the change is a symptom of L2 working, not new behaviour. The operational
lesson is the one that matters: discoveryDeferred is not evidence that
discovery was being deferred at the moment you read it. It can be arbitrarily
stale. The same latch applies to lastError (see the note at the end).

The coupling that actually cost all dispatch — main factory.ts:1983 / :2021 / :2039 / :2366

Discovery is not only the sweep. The live subscription drain is a full,
independent dispatch path that never touches the discovery lease:

#enqueueLiveEvent (:2358) → #scheduleLiveEventDrain (:2366) →
#handleLiveEventsWithYield → #handlePreparedLiveChange → #handleChange →
triageIssue (:7837) → dispatch (:7847).

Its gate at :2366 checks #liveEventDrainScheduled, #liveEventDrainActive,
#deferLiveEventDrain and #started. There is no sweep gate. It dispatches
happily while a sweep is in flight.

Except for one thing:

1983:    this.#deferLiveEventDrain = true      // before the startup backfill
2021:        const report = await this.runOnce()   // ← unbounded
2039:      this.#deferLiveEventDrain = false     // in the finally

:2039 is inside a finally around an unbounded runOnce(). A wedged
startup backfill therefore means start() never returns, #deferLiveEventDrain
stays true forever, and the live drain never starts. Both discovery paths
die together. That is why a wedged sweep meant zero dispatch instead of stale
dispatch.

This PR fixes exactly that, with no extra change. runOnce() now always
settles, so :2039 always runs, so the live drain always starts and keeps
dispatching while later sweeps are degraded. A slow sweep now costs freshness —
the outcome you asked for in B — because the durable safety net degrades while
the event-driven path stays up.

This is not a reading, it is a test. must-fire: a live daemon whose sweep is wedged still shuts down starts a real daemon whose first post-claim call never
returns. Against origin/main's factory.ts it fails after 4044 ms with
start never returned — start() itself never comes back, which is :2039
never running. With the budget, start() returns and stop() completes.

I have not removed the :3071 deferral, per your instruction, and I would
not: the lease is load-bearing for checkpoint correctness.

---

Proof

Fail-first verified by mechanism, not by assertion colour: with only
factory.ts reverted the end-to-end must-fire fails after 4038 ms with
sweep never settled — the sweep never settles, which is the production defect
in the production shape. (The must-not-fire passes with the fix reverted, as it
must: a healthy sweep is unaffected either way.)

must-fire (end to end). A sweep whose first post-claim call never returns is
aborted at its budget with the abandoned phase named; the lease handback is
observed on the store, not inferred; the next cycle runs a fresh sweep and
dispatches.

must-fire (primitive).

• Three 40 ms calls under a 120 ms budget: the third is rejected because the
  sweep is out of time, not because it is slow. This is the aggregate
  property no per-call deadline has.
• A bounded-but-always-failing call inside an unbounded retry loop ends at
  the budget after more than one attempt. That is the L3 shape.
• The signal aborts at expiry; a spent budget refuses to start new work against
  the dependency it just gave up on.

must-not-fire.

• A healthy sweep under a snug 30 s budget produces results identical to an
  unbounded control — pulled, dispatched, skipped, and the spawn list.
  Without this the trivial wrong fix (abort everything immediately) passes.
• A caller's own failure still surfaces as itself, never re-clothed as a budget
  expiry.
• With sweepBudgetMs: 0 the same hung call stays pending, so every
  rejection above is attributable to the budget and not to the wrapper.

What this does NOT cover

• It does not make anything faster and it does not find the hanging call. A
  wedged dependency still costs one whole budget per cycle.
• It does not cancel. See Mechanism above.
• The abandoned pass runs concurrently with the sweep that replaces it if it ever
  unsticks. It cannot commit a checkpoint (the store's epoch guard) but its
  in-flight side effects still land.
• start()'s own pre-backfill watermark read is outside this budget, and
  outside the orchestrator's bounds entirely. #startLiveSubscription calls
  #currentEventHighWatermark() (factory.ts:2192) before the backfill; it
  awaits mount.getEventHighWatermark() under a bare try/catch, NOT through
  #withRelayfileOperation, so relayfileOperationTimeoutMs does not reach it.
  What bounds it in production is one layer lower — the deployed client's own
  #bounded() (relayfile-cloud-mount-client.ts:1057, fix(mount): bound the relayfile change feed so a hung tail read cannot wedge the readiness sweep #368). A MountClient
  without that deadline has none here at all, and start() hangs forever.
  Found while writing the shutdown test; not fixed here because the file that
  would fix it belongs to the L2 lane this week. (Thanks to the review for
  catching that my first wording credited the orchestrator with a bound it does
  not have.)
• Shutdown is bounded by the grace window plus one teardown deadline, not by
  the budget — but a sweep whose teardown is ALSO abandoned still leaves an
  orphaned lease for one expiry window.
• The #runOnceWithReadinessDeadline abandoned-wait bookkeeping is unchanged;
  a budget expiry reaches it as an ordinary sweep failure. The seven bounded readiness reconciliation tests that cover that
  accounting now pass sweepBudgetMs: 0 — at the budget's default there is no
  abandoned-but-still-running sweep left for them to observe, which is the fix;
  0 selects the backstop underneath it, the same control idiom fix(mount): bound the relayfile change feed so a hung tail read cannot wedge the readiness sweep #368 used.
• The default changes no timing (see above).

Review fixes at this head

Every finding below was valid and is fixed, each with its own must-fire /
must-not-fire:

finding	fix
a fixed 90-min sweepBudgetMs default rejects any config that already tightened reconcileTimeoutMs, and silently caps one that loosened it	the omitted budget is derived from its SIBLING in a .transform(), never from a constant (resolvedSweepBudgetMs)
an abandoned #performRunOnce can write a stale tree into the REPLACEMENT sweep's checkpoint	#isStaleDiscoveryContinuation() compares the discoveryEnumerationPass ALS epoch — which follows the async continuation, so it carries the epoch that ISSUED the read — against the live one; applied to the checkpoint write and to overload attribution
an abandoned pass could still dispatch after teardown	the dispatch loop gets the same budget guard as the read loop
a lease claimed after the budget gave up on the claim was stranded for a full lease window	a compensating release is attached to the abandoned claim
unref'd deadline timers let Node exit before the budget fires, so a one-shot runOnce() returns without reporting the wedge or releasing the lease	both deadline timers are referenced; dispose() clears them from a finally on every path

The first one was the dangerous one: it would have taken Factory down on
deploy for any config that had tuned reconcileTimeoutMs, because the schema
throws before the daemon starts.

A second round found two more, both this PR's own lesson recurring inside it:

finding	fix
stop() outlives the sweep it started (#301), so a wedged sweep made shutdown as long as the budget — 90 min at the default	stop() arms a grace timer over the drain and then calls budget.expire(), routing the sweep into its ORDINARY abort path. NOT unref(), which is in direct tension with the P2 above: it would also let Node exit before the budget fires
the lease claim was issued before budget.run could reject a spent budget	the claim is issued inside the callback; the promise is kept outside for the late-release compensation

On shutdown: before this PR that same drain was unbounded, so shutdown on a
wedged sweep never returned at all. A sweep that starts while #stopping is
already set is expired on creation, so it cannot hand shutdown a fresh budget.

The claim-ordering finding is scoped honestly in its test: "already spent on
entry" is not reachable by construction — every path into the helper is preceded
by a budget.run that throws first, and I verified by ablation that the
integration test still passes without the fix. What makes the microtask gap
worth closing is new in the same commit: stop() can now spend a budget
asynchronously.

On the /evidence document — corrections

Three of the four readings do not survive the code, and none of them weakens the
case for this PR:

1. lastError is stale, and does not describe the wedged pass.
   #readinessReconcileLastError is cleared only on a successful pass (main
   factory.ts:2265). The wedged pass has not settled, so that breaker message
   belongs to the pass that failed at 07:51:59, not the one in flight since
   07:52:59. Same latch as discoveryDeferred.
2. lastDurationMs: 4368 is a failure's duration, not a success's. It is
   written on both paths (:2304 on failure). lastFailureAtMs 07:51:59 is
   2 ms after fleetControlPlane.lastFailureAtMs, so 4368 ms is how long the
   pass took to fail via the breaker. The last success was at 07:49:49. The
   underlying instinct is still right — 07:47–07:50 shows sub-interval sweeps —
   but this field is not the evidence for it.
3. The 07:52:59 "coincidence" is arithmetic. Both 60 s timers were armed by
   the same event: the reconcile interval (intervalMs: 60000) from the failure
   at 07:51:59.211, and the breaker's resetTimeoutMs: 60000 from
   07:51:59.209. They expire together by construction. Not a lead.
4. The 20 s roster bound is real and does fire. probe() wraps roster()
   in withTimeout (src/fleet/control-plane-circuit.ts:245-269), a genuine
   race. So the 38-minute hang is almost certainly not in roster().
   The unbounded thing on that path is the mutation after the probe
   (:180-183: "applies the local roster deadline without imposing a timeout on
   mutations") — a spawn/resume during dispatch has no deadline at all. That
   is a live L4 candidate. I have not chased it, per the brief. Under this
   PR it is a degraded sweep rather than the end of dispatch, which is the point.

Your framing stands where it counts: the failing transport is not the same one
twice, and an aggregate budget is agnostic to which one it is.

[github-actions]

@coderabbitai review

Requested for exact head 2e31da9a2801fa1cd08b4fb58e3dd7ec93aefac0.

[coderabbitai]

Warning

Review limit reached

Next included review available in 48 minutes.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: f3500919-244a-4937-98a1-f6fe2bcfa7b3

📥 Commits

Reviewing files that changed from the base of the PR and between 2c2dd86 and 1b48ff0.

📒 Files selected for processing (3)

• src/orchestrator/factory.ts
• src/orchestrator/sweep-budget.test.ts
• src/orchestrator/sweep-budget.ts

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 01452f9d-5af8-4b53-9dc0-aa2be54b3e07

📥 Commits

Reviewing files that changed from the base of the PR and between f1dc713 and 2c2dd86.

📒 Files selected for processing (6)

• src/config/schema.ts
• src/orchestrator/factory.test.ts
• src/orchestrator/factory.ts
• src/orchestrator/sweep-budget.test.ts
• src/orchestrator/sweep-budget.ts
• src/types.ts

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The PR adds configurable aggregate budgets for discovery sweeps. It applies shared deadlines across sweep phases, bounds teardown, compensates delayed lease claims, prevents stale writes, and adds unit and end-to-end coverage.

Changes

Discovery sweep budgeting

Layer / File(s)	Summary
Budget configuration and public contract src/config/schema.ts, src/types.ts	Adds sweep budget defaults, resolution, cross-field validation, and the required sweepBudgetMs live subscription option.
Shared budget and teardown runtime src/orchestrator/sweep-budget.ts	Adds shared expiry handling, cancellation signals, phase-specific errors, timer disposal, and bounded teardown deadlines.
Factory sweep enforcement and cleanup src/orchestrator/factory.ts	Applies the budget across discovery operations, lease handling, retries, commits, teardown, enumeration, dispatch, and stale asynchronous continuations.
Budget behavior and regression coverage src/orchestrator/sweep-budget.test.ts, src/orchestrator/factory.test.ts	Tests expiry, cancellation, retries, lease cleanup, shutdown, stale writes, healthy sweeps, and configurations without aggregate budgets.

Estimated code review effort: 4 (Complex) | ~60 minutes

Merge Risk: ⚪ Minimal · up to 2c2dd

The PR bounds the full sweep and teardown without an identified call-site contract issue; no actionable merge-blocking risk remains beyond normal checks and review.

Sequence Diagram(s)

sequenceDiagram
  participant Factory
  participant SweepBudget
  participant StateStore
  participant DiscoverySession
  Factory->>SweepBudget: start aggregate sweep budget
  Factory->>StateStore: claim discovery lease
  Factory->>DiscoverySession: prepare and execute discovery
  DiscoverySession-->>Factory: return discovery results
  Factory->>StateStore: checkpoint and commit results
  SweepBudget-->>Factory: signal expiry
  Factory->>StateStore: release or compensate lease

Loading

Suggested reviewers: kjgbot, miyaontherelay

Poem

A rabbit sets a deadline bright
The sweep hops through the night
Leases tidy, timers cease
Stale trails fade into peace
Fresh trees grow on schedule true

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 4 files. (2 skipped: 2 …
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly summarizes the main change: it bounds the entire orchestrator sweep instead of adding another per-call limit.
Description check	✅ Passed	The description is detailed and directly explains the aggregate sweep budget, lease handling, teardown behavior, limitations, and test coverage.

Full details: Docstring Coverage

Explanation

Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 4 files. (2 skipped: 2 too large.)

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/sweep-aggregate-budget

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

@coderabbitai review

Requested for exact head b032e85d0ce25f78c448b47b87458b733c09a086.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2e31da9a28

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[cubic-dev-ai]

Review completed against the latest diff

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[github-actions]

@coderabbitai review

Requested for exact head 5c656f98e553ce9fd29a6697cb07b61360d238a0.

[cubic-dev-ai]

All reported issues were addressed across 2 files (changes from recent commits).

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[github-actions]

@coderabbitai review

Requested for exact head 2c2dd86eecbe23d7487ca68243dd4a98e95cdbf4.

[cubic-dev-ai]

All reported issues were addressed across 4 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[github-actions]

@coderabbitai review

Requested for exact head 462aa3e28017798b5c1e6f5e6e38360f58e4b675.

[cubic-dev-ai]

All reported issues were addressed across 3 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[khaliqgant]

Scope note before anyone merges this: it bounds the stall, it does not prevent it

Recording this on the PR so the caveat travels with the change rather than living in a status thread.

Measured against the deployed worker this morning, readinessReconcile had been stalled on a single call since 09:07:32Z:

sample	inFlightMs	missedPasses	consecutiveFailures
10:14:12.170Z	3999898	66	0
10:15:57.210Z	4104938	68	0

The inFlightMs delta is 105.040s across a wall-clock delta of 105.040s — ratio 1.00000. That is one call that never returned, not a slow loop; missedPasses advances exactly one per 60s interval while consecutiveFailures stays at 0, because nothing ever fails.

The container booted at 09:03:24Z, which is after factory-cloud#78 merged, so the bounded FACTORY_STATE calls are already in that build and the stall recurred anyway — four minutes after boot.

Why that matters for this PR. The aggregate budget defaults to 90 minutes: DEFAULT_READINESS_RECONCILE_TIMEOUT_MS = 90 * 60_000 (src/config/schema.ts:62), DEFAULT_DISCOVERY_SWEEP_BUDGET_MS derives from it (:84), and resolvedSweepBudgetMs clamps the budget so it can never exceed reconcileTimeoutMs (:92). Against a stall that was 68 minutes old, this change would not yet have released it.

What it does buy is recovery, not speed. On the deployed build the reconcile timeout rejects the caller's wait and leaves runOnce() running, so every later cycle coalesces onto the wedged pass — which is exactly the discoveryDeferred: "sweep-in-flight" with missedPasses climbing one-per-60s observed above, and it never ends. This PR rejects from inside #runOnceWithDiscoveryFence, releasing the lease and clearing #runOnceInFlight so the next cycle starts clean. That converts wedged forever into wedged for at most the budget.

So: necessary, not sufficient. Please do not merge this believing it is the cure for the outage. The budget is also a config dial (sweepBudgetMs, schema.ts:131), and lowering it has a documented cost — the sweep commits its checkpoint only at the end, and #36 measured 61 minutes of cold-mirror hydration in production, so a budget under that turns a slow boot into a loop that never completes a pass. Decide that on evidence, not by reflex.

CI on this branch is currently red on package and is being worked; that is tracked separately and is not a reason to relax any assertion here.

[github-actions]

@coderabbitai review

Requested for exact head 1e9d2ec94e199b99438be78ccdb22ed031afc8cb.

[cubic-dev-ai]

All reported issues were addressed across 2 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[khaliqgant]

This PR is an outage fix — but it is inert at its default value

Flagging this before the merge decision, because the framing matters.

The outage. Factory's discovery sweep is wedged in production right now and has dispatched nothing for 39+ hours. Full measurements in #372, but in short: one sweep started 11:48:55Z and never returned, inFlightMs grows 1:1 with wall clock, missedPasses climbs one per minute, discoveryDeferred: "sweep-in-flight", and consecutiveFailures sits at 0/3 the entire time because a hang throws nothing and the breaker counts only failures. Canaries #350 and #364 have sat 39h and 20h with zero acknowledgement despite passing the deployed safety gate.

Why this PR is the right mechanism. The budget expires from inside #runOnceWithDiscoveryFence, so the sweep unwinds, the discovery lease is released, and #runOnceInFlight clears — which is precisely the coalescing that produces the absorbing state (factory.ts:2982-2988 returns the in-flight promise to every later pass). It also converts the hang into a real rejection, so consecutiveFailures finally moves, which partially closes #372 as well. And the DiscoverySweepPhase enum would name the await the sweep died on — currently unobservable, which is why the specific blocking call is still UNKNOWN.

Why merging it alone would change nothing.

• sweepBudgetMs defaults to reconcileTimeoutMs.
• DEFAULT_READINESS_RECONCILE_TIMEOUT_MS = 90 * 60_000 (schema.ts:62) — 90 minutes.
• The deployed template config/factory.config.template.json sets no reconcileTimeoutMs, reconcileIntervalMs, or sweepBudgetMs, and container-env.mjs passes no override.
• Observed container lifetime is ~50 minutes.

So the budget resolves to 90 minutes against a ~50-minute container lifetime and would never fire. Meanwhile state: 'stalled' is declared at 10 minutes (READINESS_RECONCILE_STALL_INTERVALS = 10, public-health.ts:64/491).

Therefore: this is an outage fix only if it lands together with an explicit sweepBudgetMs set near the 10-minute stall threshold. As a bare merge it is routine and production stays wedged. That distinction belongs in the merge decision.

CI is currently red / UNSTABLE on this branch; a lane is discriminating flake from real failure (note the known flakes #373 and #342 both touch the failing suites). I am not merging — the merge gate is the principal's.

[github-actions]

@coderabbitai review

Requested for exact head 1b48ff0238656f7339dd62b1cc0c2755031133c4.

[khaliqgant]

Corrections to my comment above, and a status change

Four things in my earlier comment need correcting. A lane verified my claims independently and disputed two of them; it was right, and I would rather correct them here than have anyone act on the originals.

1. CI is no longer red — this PR is green and merge-ready

I said "CI is currently red / UNSTABLE". That is now stale. At head 1b48ff0: CI: success, all five jobs green, and the prior head 1e9d2ec was fully green too — two consecutive green runs. mergeStateStatus: CLEAN, mergeable: MERGEABLE, 0 of 15 review threads unresolved.

The earlier red at 462aa3e was two pre-existing flakes, proven on a control ref that predates this PR: teammate-mcp.test.ts (#373) and factory.test.ts slackReplyRoutesFencedDuringDrain (#342). The strongest single piece of evidence is CI run 32821385637 on main at 952d450e — this PR's exact merge-base — failing with the identical teammate-mcp.test.ts failure on a ref containing none of this change. The counter string in the second failure appears zero times in this diff.

2. My "set it near the 10-minute stall threshold" recommendation was risky — prefer ~15 min

This is the correction that matters most, because it is the one someone might have acted on.

schema.ts:76-82 documents that a sweep budget set below realistic cold-mirror hydration converts a slow boot into a loop that never makes progress — the sweep checkpoints only at the end, so it restarts from zero each time. #36 measured 61 minutes for that hydration in production, and the budget covers startup backfill too. A 10-minute budget sits uncomfortably close to that cliff, and the failure mode it risks — a boot loop — is worse than the stall it fixes.

Recommended instead:

"liveSubscription": { "sweepBudgetMs": 900000 }

resolvedSweepBudgetMs = min(900000, reconcileTimeoutMs) = 15 min, so this single key is sufficient and reconcileTimeoutMs keeps its 90-minute envelope. 15 min sits three orders of magnitude above the observed warm sweep (lastDurationMs: 917 — under one second), below container lifetime so it can actually fire, and further from the #36 cliff than 10 min.

3. container-env.mjs does not exist — I named a file that isn't there

I wrote that "container-env.mjs passes no override". No such file exists at scripts/ or anywhere in the org by filename search. I should not have stated it.

The substance still holds, by better evidence: an org-wide code search returns 0 hits for sweepBudgetMs, liveSubscription, reconcileTimeoutMs and reconcileIntervalMs in factory-cloud, the deployed template has no liveSubscription block at all, and scripts/render-config.mjs injects none. Corroborated live — /healthz reports intervalMs: 60000, exactly the schema default.

4. The "~50 minute container lifetime" is not independently measured

I stated ~50 min as though it were established. It rests on an inference (repeated dependency-park comments implying fresh processes), and a direct check found one container alive at 34 minutes and still running. One sample is not a distribution. Treat the lifetime as unknown; the argument does not depend on the exact figure, only on it being well under 90 minutes, which is not in doubt.

Also: the coalescing site is factory.ts:3056-3059 at current head — my 2982-2988 was the pre-PR ref.

What stands unchanged

Merging this alone still does not fix the production wedge. The budget resolves to 90 minutes with no config override, so nothing engages. The PR makes the wedge recoverable in principle and nameable in practice (its DiscoverySweepPhase enum would identify the await the sweep dies on — currently UNKNOWN), but the config change has to ship with it. The wedge is still live: latest samples show inFlightMs past 1.8M and missedPasses at 30, consecutiveFailures still 0/3.

Merge gate remains the principal's; I have not merged.