Skip to content

feat(runtime): meter codex turns and add a fail-closed wake fuse - #4

Open
apresmoi wants to merge 5 commits into
mainfrom
feat/codex-turn-usage
Open

feat(runtime): meter codex turns and add a fail-closed wake fuse#4
apresmoi wants to merge 5 commits into
mainfrom
feat/codex-turn-usage

Conversation

@apresmoi

Copy link
Copy Markdown
Collaborator

Scope

Stacked branch. This branch is cut from the local, unpushed
feat/operational-ecosystem, so the diff against main carries 24 commits that
are not this change. This PR's own range is the last three commits:

  • ed8f8bb feat(runtime): meter codex turns into the per-turn usage ledger
  • 8cf7a1f feat(runtime): add a fail-closed wake fuse with a named halt point
  • a64e286 fix(pi): publish a paid codex turn whose stream is truncated or trailing

Why

An organization of 16 agents exhausted a paid subscription in 24 hours across
2,481 wakes. Two things had to exist before that organization could be run again
and measured: every engine's spend had to be visible, and something had to be
able to stop it.

Codex metering

Codex turns were completely uninstrumented — TURN_USAGE_ENGINES was
["agy", "grok"] and Codex ran in plain-text mode, which reports no token
accounting at all. Codex now runs codex exec --json and a new decoder reads
the turn.completed usage frame.

The arithmetic is verified against a live codex-cli 0.151.0 capture:
total_tokens 18115 = input 18110 + output 5, so cached_input_tokens is a
subset of input_tokens — the opposite of AGY's disjoint cache_read_tokens.
Copying AGY's formula would over-count every cached turn, and a test locks that.

Codex's output format is now a Daimon-owned boundary like AGY's, guarded by
assertSafeCodexCommandArgs.

The wake fuse

A durable, fail-closed admission ledger with a named halt point. On a trip:

  • turns already running are allowed to finish — the money is already spent;
  • deliveries queued at accepted are driven terminal, because leaving them
    parked is the burnout: startup re-dispatches every accepted record and the
    bus polls non-terminal records forever;
  • deliveries arriving are refused through the existing stopped /
    host_stopping response.

No new wire enum, so DAIMON_CONTRACT_MANIFEST_SHA256 is unchanged and every
pinned image still attests.

Operator stop:

docker exec <container> touch /var/lib/spawnfile/daimon/usage/fuse.stop

Review

Reviewed adversarially by a different vendor's model than the implementer.
Three P1s were found against the decoder — a paid turn's reply was discarded
when a frame followed turn.completed, when the stream was truncated, and when
a blank message clobbered a good one — and all three are fixed in a64e286
with failing-before tests. Deferred P2/P3 findings are recorded in
WAKE_WORK_LEDGER.md.

Gates

  • npm test — 439 tests, 435 pass, 0 fail
  • npm run typecheck — clean
  • npm run verify:contract-manifest — digest unchanged

🤖 Generated with Claude Code

@apresmoi

Copy link
Copy Markdown
Collaborator Author

Two further rounds landed since the PR description was written:

  • ef0db4f fixes two P0s found by an independent adversarial review — a ceiling trip was memory-only (so a restart replayed the whole backlog as paid wakes, the exact burnout the fuse exists to prevent), and tripFuse took a single snapshot so a persist landing after it left a record parked at accepted forever. Also four P1s: the token ceiling read a hard-coded path while writers honour DAIMON_TURN_USAGE_LEDGER_PATH; POST /v1/wake bypassed the fuse entirely; DAIMON_WAKE_FUSE=off was silent; and the operator poll's void tripFuse could take the process down on an unhandled rejection.
  • 79e74af fixes a P1 found by running the mutation checks directly: trip() set this.reason before writing the durable marker, so a failed marker write left the fuse tripped in memory with nothing ever retrying — P0-1 reopening through its own error path.

Every fuse guarantee is now mutation-verified — deleted, suite observed red, restored:

guarantee deleted result
trip-marker write red
trip-marker restore in open() red
start() fuse gate before recoverable() red
terminalization loop beyond one pass red
resolveTurnUsageLedgerPath in the token sum red
wake() gate after a trip red
stopping re-check between admit and persist red (was green; fixed in 79e74af)

npm test 450 tests / 446 pass / 0 fail; verify:contract-manifest unchanged.

Deferred P2/P3 findings are listed in WAKE_WORK_LEDGER.md rather than fixed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant