Skip to content

Make session boot and readiness observable, and stop a failed submit from wedging the pane - #627

Merged
Juliusolsson05 merged 6 commits into
mainfrom
feat/session-lifecycle-observability
Jul 28, 2026
Merged

Make session boot and readiness observable, and stop a failed submit from wedging the pane#627
Juliusolsson05 merged 6 commits into
mainfrom
feat/session-lifecycle-observability

Conversation

@Juliusolsson05

Copy link
Copy Markdown
Owner

Why

Agent boot has been patched ~30 times since 2026-04-11 without converging. docs/decomposition/agent-boot-readiness.md establishes why, with measured evidence: no boot event has ever been recorded. Every one of those fixes was authored from source reading against a failure nobody had captured.

Concretely, before this PR:

  • AppRunIncidentKind has 20 event kinds — crash, heap, window, MCP, orchestration. None covers spawn, recover, adopt, or readiness.
  • The perf tracer is enabled: false behind AGENT_CODE_PERF, and measures durations, not gate state.
  • The session recorder arms on a session's first event — the boot window is over before it starts.
  • publishPromptGate is edge-triggered: a session that never becomes ready emits nothing after the initial {ready:false, reason:'starting'}. The renderer then waits forever with no further fact.
  • The user-facing text is agent failed to start, with no reason, phase, or elapsed.

This PR implements Stages 1 and 2 of that decomposition. It does not fix why backends go missing — it makes the next occurrence diagnosable and non-wedging. A PR that also quietly fixed the root cause would be patch #31 and would invalidate the corpus it exists to produce.

What ships

A session.lifecycle event stream, always on, riding the existing AppRunJournal (already byte-capped at 50 MiB/run, redacting, and degrades to a no-op on an unwritable ~/.config). ~20 emit points across recover / spawn / provider start / readiness / wake / history load / submit / delivery / kill.

npm run lifecycle:summarize — one phase ladder per session, flagging stranded claims, transcript loads that never ended, sessions that never became ready, long gate stalls, classified delivery rejections, and wake storms.

The pane names its stall. waiting for agent · 1m 34s instead of starting agent. Elapsed time is the point: it's what separates wedged from normal. Failed panes append their typed recovery code, because ownership-conflict and start-failed need different responses.

A failed submit no longer wedges the pane. Reported as Cannot deliver prompt: <id> is not a live agent session, followed by Sending · 17s counting up forever until the agent was reloaded.

The two defects behind the report

A — registry split-brain (recorded, not fixed). sessionManager.ts finds no entry for a session the renderer believes is live. delivery.reject now classifies it as never-owned (renderer invented or resurrected an id) vs entry-lost-after-owned (a teardown the renderer never observed). Two different defects, byte-identical to the user today.

B — the optimistic submit state was never unwound (fixed). setStreamingBaseline sets streamPhase: 'submitting' before the attempt; the catch recorded the failure but never touched the phase. Nothing else could — the three paths back to 'idle' are an exit event (main has no entry to exit), emptyRuntime() (that is the reload), and a provider semantic event (nothing was written, and streamPhaseMachine refuses to stomp submitting anyway).

Why B is not the conditional trap this subsystem keeps falling into

#548's kill-timeout became #596; TileLeaf's !inputReady gate became #598. Both were guards that inferred state. This infers nothing: it unwinds only when main reports promptWritten === false && enterWritten === false. Nothing written ⇒ no turn can start ⇒ the phase is provably stale.

The uncertain case (something was written) is deliberately untouched — a turn may genuinely be running. The streamPhaseMachine guard is not relaxed; it's a shipped regression's tombstone.

What keeps this from becoming the mess it diagnoses

Nothing in production reads these events. The journal is a sink, never a decider. No new if statements, no new state, no awaits, removable in one revert. ~230 lines against a 23,459-line surface.

The vocabulary is closed and payload keys are an allowlist — unknown keys are dropped, so a careless call site loses a field instead of leaking one.

Trap documented in events.ts: the journal's redactor silently drops any top-level key matching /prompt|content|text|env|token|secret|key/i. A field named promptWritten or context would have recorded nothing while looking correct in review. Every allowlisted key is checked against that regex.

Notable findings

  • The compiler found 13 wake call sites, not the nine a grep suggested. Making caller a required parameter of ensureSessionLive turned the type system into the census. All 13 are instrumented in place, not consolidated — which of them actually differ is a corpus question, and merging them on a guess is exactly what shipped Terminal-mode Claude agents are killed 30s after any pane remount (Spotlight, Reader, Settings, tab switch) #596 and Answer provider conditions without the composer's readiness gate #598.
  • gate.eval recovers information that was being destroyed: publishPromptGate collapses replay-pending / composer-unpainted / human-draft into provider-not-ready before it leaves the provider. On-change plus a 5s sampler, because an unconditional emit off screen snapshots would be a 60 Hz firehose that buries the boot breadcrumbs inside the journal's own byte ceiling. The sampler stops itself when nothing is stalled.
  • The three pre-existing session.recovery.* events are migrated into the closed vocabulary rather than run alongside it — they had no readers.

Known gaps, stated deliberately

  • The pane shows the coarse readiness reason. The detailed verdict is collapsed before it leaves main; recovering it means widening the SessionInputReadiness contract — Tier 3 transport this PR does not touch. It's in gate.eval meanwhile.
  • Codex and opencode declare prompt-gate but never emit it. They latch a coarse boolean and genuinely cannot say why they aren't ready. That silence is left visible rather than faked.

Verification

  • npm run typecheck — clean (tsc -b, both projects)
  • npm run test:contract — satisfied
  • npm run check:keybindings — OK
  • npm test1788/1789

The one failure is store.test.ts (prompt-template migration), a 5s-timeout test unrelated to this change. Confirmed pre-existing: origin/main was run three times and produced 1 failure, then 3, then 0, in the same two timeout-sensitive files. It passes in isolation on both branches.

Not verified by launching the app; correctness is argued from source and pinned by tests.

Next

Stage 3 is not an engineering task — it's using the app until the corpus has repeats. Then npm run lifecycle:summarize feeds the Stage 4 catalog, and fixes start one shape at a time, each with a fixture.

🤖 Generated with Claude Code

Juliusolsson05 and others added 6 commits July 28, 2026 17:41
Agent boot has been patched ~30 times since 2026-04-11 without converging.
docs/decomposition/agent-boot-readiness.md establishes why with measured
evidence: no boot event has ever been recorded. Every fix was authored from
source reading against a failure nobody had captured.

This is stage 1 of that decomposition, main-process half.

WHY it rides AppRunJournal rather than a new store: the journal already solves
always-on disk logging, and each solution is scar tissue we must not re-earn —
a per-run 50 MiB ceiling, a drop-oldest pending bound, redaction, and a silent
degrade to no-op on an unwritable ~/.config so diagnostics can never brick
launch. Co-locating also puts a boot stall in the same events.jsonl as heap
pressure and crash breadcrumbs, one scan from being correlated.

WHY the vocabulary is closed and the payload keys are an allowlist: an open
event name rots into noise nobody reads, and an open payload is how an
always-on stream becomes a privacy incident. Unknown keys are dropped, so a
careless call site loses a field instead of leaking one.

WHY nothing reads these events: the journal is a sink, never a decider. That is
what keeps ~15 emit points from constituting new coupling — no control flow
depends on them and the subsystem is removable in one revert.

Notable trap recorded in events.ts: the journal's sanitizer silently drops any
top-level key matching /prompt|content|text|env|token|secret|key/i, so
`promptWritten` and `context` would have recorded nothing while looking correct
in review. Every allowlisted key is checked against that regex.

The three pre-existing session.recovery.* names are migrated into the closed
vocabulary rather than run alongside it; they had no readers.

Two emit points target the reported failure directly. delivery.reject fires on
"Cannot deliver prompt: <id> is not a live agent session" and classifies it as
never-owned (renderer invented or resurrected an id) versus
entry-lost-after-owned (a teardown the renderer never observed) — two different
defects that are byte-identical to the user today. recover.adopted records the
adopted backend's readiness, the distinction #596 turned on.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Main sees ownership; it is structurally blind to intent. The nine wake call
sites, rehydrate's completion accounting, transcript loading, and composer
submit all live in the renderer, so a ladder assembled from main alone cannot
answer "why did something try to wake this pane" — the question #596 and #598
both turned on.

WHY `caller` is a required parameter of ensureSessionLive rather than an
optional tag: an untagged wake.request says a wake happened, which we already
knew. The tag is the entire diagnostic value. Making it required also turned the
compiler into the census — and it found THIRTEEN call sites, not the nine a grep
had suggested. WAKE_CALLERS is corrected to match, and the sites are
instrumented in place rather than consolidated, because which of them actually
differ is a question for the corpus, not a guess.

history.load start/end brackets the #283 class directly. `status` separates
'no-terminal-write' (marked-but-never-loaded) from 'dropped-ready'/'dropped-error'
(the write ran but its runtime key was gone) — two distinct defects that both
presented as a pane spinning on "loading transcript" until a manual reload.

submit.begin/result records the ordering behind the stuck-`Sending` bug:
submit.begin, then the optimistic streamPhase, then a failed result with nothing
following it. bodyWritten/enterWritten are named to survive the journal's
redactor — a field called promptWritten would have recorded nothing while
looking correct in review.

report.suppressed gets its own name rather than folding the count onto a nearby
event: a reader must be able to tell "this pane emitted nothing" from "this
pane's events were dropped", and corrupting a ladder to save a name would defeat
the analysis the stream exists for.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
publishPromptGate reduces its detailed verdict to 'ready' | 'provider-not-ready'
before it leaves the provider, so by the time readiness reaches main the
difference between "replaying history", "the composer never painted", and "a
human has a draft in the box" is already gone. Those are three different
problems and they are indistinguishable in every log we have.

SessionManager now listens to the `prompt-gate` event Claude ALREADY emits. No
gate logic changes anywhere — this is a listener, not a decision.

WHY on-change plus a 5s sampler rather than every evaluation:
derivePromptGateState runs off screen snapshots, so an unconditional emit would
be a 60 Hz firehose that buries the boot breadcrumbs inside the journal's own
byte ceiling — the instrumentation evicting the evidence it exists to keep.
On-change plus periodic sampling answers "stuck at composer-unpainted for 90s"
in a few hundred bytes instead of a few megabytes, and the sampler stops itself
once nothing is stalled so an idle workspace writes nothing at all.

`elapsedMs` is the point. A single not-ready event is worthless — that is the
normal state for a moment during every boot. The same verdict still holding 90
seconds later is the whole bug, and nothing in the app recorded time-in-state.

Codex and opencode declare `prompt-gate` without emitting it: they latch a
coarse boolean and genuinely cannot say why they are not ready. Declaring the
key keeps them satisfying AgentSession (the interface-merging shape the legacy
Claude condition events already use) while the silence stays honest.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…itten

Reported as: "Cannot deliver prompt: <id> is not a live agent session", after
which the pane shows `Sending · 17s` counting up forever and only reloading the
agent clears it.

setStreamingBaseline sets streamPhase 'submitting' BEFORE the delivery attempt.
When delivery failed, the catch recorded the failure and showed a toast but
never touched the phase — and nothing else could. There are exactly three paths
back to 'idle' and under a before-write failure none of them can fire:

  1. onSessionExit needs a real exit event; main holds no registry entry to
     exit, which IS the failure.
  2. emptyRuntime() only runs for a fresh runtime — that is the agent reload,
     and it is why reloading was the only escape.
  3. reduceStreamPhase needs a provider semantic event; nothing was written so
     none will arrive, and streamPhaseMachine deliberately refuses to stomp
     'submitting' from screen-derived signals regardless.

WHY this is not the conditional trap this subsystem keeps falling into: #548's
kill-timeout became #596 and TileLeaf's !inputReady gate became #598, and both
were guards that INFERRED state. This infers nothing. It unwinds only when main
REPORTS that neither the body nor Enter reached the provider. Nothing written
means no turn can start, so the optimistic phase is provably stale rather than
probably stale. The `uncertain` case — something WAS written — is deliberately
untouched, because a turn may genuinely be running and unwinding could hide it.

The streamPhaseMachine guard is NOT relaxed. It is a shipped regression's
tombstone; the repair belongs at the site that owns the optimistic write.

Also in this commit, the pane stops lying about why it is not accepting input.
Every non-ready state used to render 'starting agent', so a wedged pane and a
healthy one looked identical — a large part of why "the agent takes minutes to
start" was never actionable. It now names the reason and, more importantly,
times it: "waiting for agent · 1m 34s" is evidence, "starting agent" is not. A
failed pane appends its typed recovery code, because 'ownership-conflict' and
'start-failed' need completely different responses from the user.

Known gap, deliberate: the provider's DETAILED verdict (replay-pending vs
composer-unpainted vs human-draft) is collapsed to 'provider-not-ready' before
it leaves main, so the pane cannot show it yet. Recovering it means widening
the SessionInputReadiness contract — Tier 3 transport this PR does not touch.
The detail is in the lifecycle journal's gate.eval meanwhile.

WorkIndicator's private 1 Hz elapsed hook is extracted to
@renderer/lib/useElapsedSeconds rather than copied: two ticking hooks is how
one drifts to a different interval and the UI starts disagreeing with itself
about how long something has been happening.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The lifecycle stream is only worth recording if a human can read it without a
JSONL viewer and a lot of patience. Stage 3 of the decomposition is "use the app
normally for a week and collect real boots", which is only tolerable if reading
the result is one command.

`npm run lifecycle:summarize` prints one phase ladder per session with per-rung
offsets, and flags the shapes the decomposition names: a stranded recovery claim
(ownership taken and never resolved), a transcript load that never ended (#283),
a session whose only readiness fact is the seeded 'starting' (the "takes minutes
to start" fingerprint), a long gate stall with its reason, a classified delivery
rejection, a fired submit unwind, and a wake storm (many wakes collapsing onto
few recoveries — the #596 remount shape).

`--dir` reads runs from anywhere, so a journal copied off another machine or out
of a debug bundle can be summarized without moving files into ~/.config — and so
the script is verifiable against a fixture without touching real user data.

Diagnoses are deliberately conservative: a flag means "look at this", never
"this is the bug". The corpus decides what is real, not this script.

The decomposition doc is updated in place rather than appended to, per its own
convention: Stages 1 and 2 are marked shipped, Stage 2's known gap is stated
(the pane shows the coarse readiness reason because the detailed verdict is
collapsed before it leaves main, and widening that contract is a Stage 4
decision made from evidence), and §3's wake-site count is corrected from nine to
thirteen — the compiler found four the grep had missed, which is a small
instance of the document's own thesis.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two orchestrated reviewers (bloat/test-quality and correctness) found 17
findings between them. Every claim below was verified against source before
acting on it.

THE OWNER'S CONCERN WAS CORRECT. streamingUnwind.test.ts contained three tests
of a predicate defined inside the test file — they imported nothing from
production and asserted `!false && !false === true`. Worse, they papered over a
gap: unwindStreamingBaseline, the actual repair, had ZERO coverage, and the
renderer test this PR's own plan promised was never written. The tautologies are
deleted and streamingUnwind.renderer.test.tsx now drives the real hook.

CORRECTNESS DEFECTS FIXED

The unwind did not cover Codex AT ALL. Codex never goes through deliverPrompt —
its submit is a sequence of raw `send` calls throwing plain Errors with no write
evidence, so `failed` was always null and the guard never fired. A Codex pane
with a dead backend reproduced the reported bug byte for byte. Fixed by counting
sends that returned successfully; a blanket unwind would have been wrong because
a bracketed paste issues several writes and only a later one may throw.

Every healthy pane mounted a permanent 1 Hz timer. inputReadinessChangedAt is
non-null for a READY pane too, so useElapsedSeconds ticked forever while
resolveReadinessText returned null and nothing was displayed — fifteen panes,
fifteen idle timers, fifteen re-renders a second. The text is now resolved
without a clock first, which answers "is anything shown" before any timer
mounts. That is the invariant useElapsedSeconds documents and the first version
violated.

The elapsed number could invert. Only one of six readiness-write paths stamped
the clock, so a pane ready for forty minutes that was then woken rendered
"starting agent · 40m 12s" on a backend two seconds old — the number whose only
job is separating normal from wedged. All paths now restamp.

The transcript line counted the wrong clock entirely: it read the readiness
timestamp, so a pane whose readiness settled ten minutes ago showed "loading
transcript · 10m" the instant a load began. Fabricated evidence in the one place
(#283) where duration IS the diagnostic. transcriptStatusChangedAt added.

The gate sampler could blind the incident journal. It re-recorded every
non-ready gate every 5s forever — including `occupied` (a human draft) and
`blocked` (an unanswered prompt), which are the app working correctly. A parked
trust dialog emitted ~17,280 events/day, and reserveJournalBytes is a PERMANENT
latch: once a run hits 50 MiB every later event AND incident is dropped, crash
breadcrumbs included. Now skips waiting-on-a-human states and stops after ~2
minutes of stall, when the signal has saturated.

gate.eval's dedup key dropped the field that distinguishes blocked states:
`blocked` carries `condition`, not `reason`, so trust-dialog → permission-prompt
compared equal and was silently dropped — in the one event whose stated purpose
is recording what a gate is blocked on. It also survived id reuse, letting a
fresh backend inherit a dead one's `since` and skip arming the sampler.

wake.result reported ok:true at the recovery boundary, but recovery succeeding
is not the wake succeeding — the readiness wait can still time out, kill the
backend and fail the pane. The #548/#596 path was being journaled as a success.

Three WHY comments stated the opposite of what the code does: publishPromptGate
already dedupes by JSON.stringify, so gate.eval records transitions and the
"60 Hz firehose" it claimed to prevent was impossible. In a repo whose comment
policy is load-bearing, an inverted WHY is a defect.

Also: submit.begin overloaded `ok` to mean "has images" (every image submit
would have bucketed as a failure); the two AgentTerminalLeaf wake tags were
swapped, so the two most similar sites carried each other's meaning; spawn.end
duplicated recover.spawned's duration exactly and is deleted; three data keys
had no emitter.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Juliusolsson05
Juliusolsson05 merged commit 428aed2 into main Jul 28, 2026
1 check passed
@Juliusolsson05
Juliusolsson05 deleted the feat/session-lifecycle-observability branch July 28, 2026 18:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant