A small, readable agent framework built on the Claude API — the agentic loop, a tool
registry, a permission system, and session persistence — with four ways to drive it: a
terminal REPL (or -p, with JSON output for scripts), a web UI, the Agent Client
Protocol for an editor, and a library API.
It is deliberately not a wrapper around someone else's agent SDK. The loop, the tool
protocol, and the permission model are all in src/, about 5,300 lines of TypeScript.
CLI (REPL, -p) ─┐
Web UI ─┤
ACP (--acp) ─┼─► Agent ─► ModelClient ─► Claude API, or any OpenAI-compatible API
Library ─┘ │
├─ ModelRouter optional — picks the tier once per session
├─ ToolRegistry Bash · Read · Write · Edit · Glob · Grep · WebFetch
│ · Task · TodoWrite · Skill · MCP servers' tools
├─ PermissionSystem rules per tool or argument; a deny always wins
│ ├─ Hooks Claude Code's format — PreToolUse, PermissionRequest, …
│ └─ RiskGate optional — clears the easy Bash "ask" cases
├─ Context AGENTS.md, skills, prompt caching, compaction
└─ SessionManager saved every turn, resumable
npm install
cp .env.example .env # add your ANTHROPIC_API_KEY
npm run cliThe risk gate (below) is the one part that needs a second provider, because it reads token probabilities and the Anthropic Messages API does not return them. A local Ollama does, needs no key, and costs nothing:
ollama pull llama3.1:8b
AGENT_JUDGE_API_KEY=ollama AGENT_JUDGE_BASE_URL=http://localhost:11434/v1 AGENT_JUDGE_MODEL=llama3.1:8b npm run cli -- --ask --gateWithout it, --gate allowlist is offline and needs nothing — it just clears
less. npm run eval:risk-gate runs on the allow-list and needs no setup at
all, so its rows are reproducible from a clean clone; the llm rows need a
judge standing up first.
The gate in a real session, in both directions:
› Run exactly: wc -l src/agent.ts
Risk gate allowed Bash — worst P=0.074 (exfiltrates) < 0.2
⚙ Bash — wc -l src/agent.ts ok in 88ms
506
› Clean the build. Run exactly: rm -rf dist
Risk gate deferred Bash — P(destroys-data)=0.995 is not below 0.2
⚠ Permission required for Bash
Allow? [y/N/a (always)/d (deny always)]: n
› Use rmdir /s /q dist instead
Risk gate deferred Bash — P(destroys-data)=0.817 is not below 0.2
⚠ Permission required for Bash
The third exchange is the one worth having. After denying rm -rf dist I asked for the
Windows equivalent by hand, and the judge deferred that too, at 0.817 — a command the
allow-list models not at all and would have had nothing to say about.
One-shot mode, for scripts and pipes:
npm run cli -- -p "summarise the README" --read-onlyThe web UI is the same framework behind an Express server with SSE streaming:
npm run server # :3001
npm run client # :5174The gate runs there too, and the browser is where its behaviour is easiest to see: a cleared call carries the probability it cleared on, and a deferred one becomes a card with the judge's own reasoning on it.
The server runs tools on this machine and has no login, so it only answers this machine:
it listens on 127.0.0.1, and refuses any request whose Host or Origin is not a
loopback address — another website's page, or one reached by DNS rebinding.
Beyond that the server is only a transport. Each chat message is one Agent run: its
events go out as SSE, its permission prompts come back as POST /api/permission, and
Stop or closing the tab aborts it. Which API it calls is the browser's choice — the
provider presets set it, and for a custom base URL so does API format in Settings,
since MiniMax or a local Ollama serve both and the URL does not say which.
wc -l src/agent.ts scored 0.065 and ran — the CLEARED tag is the only trace in the
transcript, because the gate's entire effect is a prompt that does not appear. The rail
on the right keeps the rest: every call the gate was asked about, all four of its
answers, and how long the judge took.
rm -rf dist scored 0.994 and stopped. The card names the command rather than the tool,
since "Bash" is not a decision anyone can make and rm -rf dist is, and it shows what
deferred it: a call held at 0.21 deserves a different glance from one held at 0.994.
The CLI transcript further up scored 0.995 on that same command in a different session. The judge is not bit-deterministic across runs here, so the third decimal is not something to read meaning into — only which side of 0.20 it lands on.
The client has three themes, switched from the theme button or Settings. They share the components but not the layout. Instrument, above, is dark and built around the numbers. Editorial sets the transcript like a page: each question as a pull quote, each tool call as a numbered margin note holding the gate's four answers, and a held call as a notice with the number that stopped it set large:
Aurora is one glass column over a slow gradient, with every gated call drawn as a ring filled to its worst answer:
All four images come from one real run against a real judge — the loop and the judge
both llama3.1:8b on a local Ollama — and docs/ is regenerated by driving the live UI
with docs/capture-screenshots.mjs, not by mocking the props.
Around a reply, the transcript also shows what the loop did beyond its own tool calls: the task list TodoWrite keeps, as a checklist; a note when the conversation was compacted, and at how many prompt tokens; and a line for each tool a subagent ran under its Task call. Settings switches tools on and off, the agent's own TodoWrite, Task and Skill among them; a tool switched off there is never offered to the model.
Or use it as a library:
import { Agent } from "agent-app"; // the package name in package.json; not published to npm
const agent = new Agent({
model: "claude-opus-5-5",
allowedTools: ["Read", "Glob", "Grep"],
});
const result = await agent.run("Explain what this codebase does");
console.log(result.text, result.usage.estimatedCostUsd);| Flag | |
|---|---|
-p, --print <prompt> |
run one prompt, print, exit: 0 if the model finished, 2 if the run stopped short, 1 on an error, 130 if aborted |
--output-format <f> |
with -p: text, json (one result object) or stream-json (an event per line, then the result); asks are denied, since nobody is there to answer |
-m, --model <id> |
default claude-opus-5-5 |
-C, --cwd <path> |
working directory for file and shell tools |
--resume <id> |
continue a saved session |
--allow-all / --ask / --read-only |
permission preset (default --ask) |
--allow <rule> / --deny <rule> |
e.g. "Bash(npm test *)", "Read(~/.ssh/**)"; repeatable, a deny always wins |
--hooks <file> |
lifecycle hooks, in Claude Code's settings format |
--mcp-config <file> |
MCP servers, in Claude Code's .mcp.json format |
--acp |
serve the Agent Client Protocol on stdio, for an editor or a harness |
--gate [backend] |
score the ask cases: llm (default) or allowlist (offline) |
--gate-threshold <n> |
auto-allow below this P; default 0.20, model-specific |
--cheap-model <id> |
route each new session between this and --model; needs --gate |
--effort <level> |
low … max, sent as output_config.effort; unset, the model's own default |
--compact-at <n> |
compact once a prompt reaches n tokens, or off; default 80% of the window, ≤150K |
In the REPL: /help /tools /cost /sessions /resume <id> /new /model [id]
/permissions <preset> /gate [backend] /cwd [path] /exit. Ctrl+C stops the run in
progress — calls already running finish and the session is saved — and a second one quits.
The loop (src/agent.ts) sends a prompt, executes any tool_use blocks the model
returns, feeds the results back, and repeats until the model stops asking for tools or
maxTurns runs out. Calls that change nothing — Read, Glob, Grep, WebFetch — run
concurrently, capped by AGENT_MAX_CONCURRENT_TOOLS; Bash, Write and Edit wait for what
came before them and run one at a time, in the order asked. Permission prompts queue, so
the user is asked one thing at a time. Each call's input is checked against its tool's
schema before anything else, and each result the model sees is capped at 40,000
characters (AGENT_MAX_TOOL_OUTPUT), start and end kept; the whole text is saved to a
file the cut names, so the middle is one Read away (Read itself says to use offset). A reply that max_tokens cuts off
in the middle of a tool call is asked again with twice the room, up to 64,000; one that
stays cut off, or ends in a refusal, is neither run nor saved, since a tool_use without
its result makes every later request fail. run(prompt, { signal }) can be aborted, and
saves the session up to that point. Consumers subscribe with agent.on(event => …) and
get session, tool_request → tool_denied or tool_start → tool_end (all keyed by
the call's id), turn_start, turn_retry, turn_end and done, plus text_delta /
thinking_delta when stream: true.
The model (src/model/) is behind a ModelClient: AnthropicClient by default,
OpenAICompatibleClient for any Chat Completions endpoint, or a scripted one in tests.
History stays Anthropic-shaped throughout, and a client for another API converts at its
own edge, so the loop never branches on provider. With caching on, the Anthropic client sets two cache
breakpoints, on the system prompt (which covers the tools) and on the conversation's last
block, so each turn reads the history before it from cache instead of paying for it again.
The system prompt holds nothing that changes within a session: the working directory and
the date are appended to the conversation when they change. Reasoning an OpenAI-compatible endpoint streams
back (reasoning_content, or reasoning) is kept in the history and returned on the next
request, which DeepSeek's thinking mode requires once tools are involved; the Anthropic
client leaves it out.
Tools (src/tools/) subclass Tool<T>, declaring a JSON Schema and a summarize()
used for permission prompts. ToolRegistry resolves the per-run set from allowedTools
and disallowedTools. Bash runs in bash — Git Bash on Windows, cmd.exe only when there is
none, and the tool's description tells the model which (AGENT_SHELL names another).
Output that is not UTF-8 is decoded with the console's code page, line by line.
Permissions (src/permissions/) resolve each call to allow, ask, or deny, with
presets for read-only and ask-before-dangerous (Bash, Write, Edit and WebFetch). Rules use
Claude Code's syntax — Bash(npm test *), Read(~/.ssh/**) (which covers Glob and Grep
too), Edit(src/**) (and Write), WebFetch(domain:docs.python.org) — through parseRule
or --allow / --deny. A matching deny always wins; otherwise the most specific rule
does, a pattern over a tool over *. An allow rule matches only a simple command, so
allowing npm test * does not allow what follows an &&, while a deny matches any part of
a compound one. ask goes
through an injectable PermissionPrompt, so the caller decides how to reach the user —
the CLI reuses its own line reader, and the server sends the question out over the SSE
stream and parks the tool call on a promise until a separate POST /api/permission
answers it. That second path is why the prompt is injectable at all; until recently the
server ran defaultMode: "allow" and executed every tool call without asking, which was
the one configuration the CLI never offered. It fails closed on a timeout and on the tab
closing.
TodoWrite keeps the model's own task list for work with several steps: rewritten whole on each call, at most one item in progress, saved with the session, and shown as a checklist in the CLI and under the reply in the web UI.
Subagents: a Task tool hands a self-contained task to a subagent with a fresh
context and returns only its final answer, so a search across many files costs the
conversation one result. general-purpose has every tool but Task; subagents in the
config adds types with their own prompt, tools and model. A subagent shares the parent's
client and permission system — its calls are asked about in the same queue, under the same
rules — its usage counts toward the parent's, and it runs one at a time, since two editing
the same files at once would be the race the scheduling exists to prevent. The CLI shows
its tool calls indented under the Task call, and the web UI a line for each.
Skills (src/context/skills.ts) follow the Agent Skills standard: a folder with a
SKILL.md whose frontmatter gives a name and a description, under .agents/skills,
.claude/skills or .agent-app/skills, in the project first and then the home directory.
A new session is told each skill's name and description, a line apiece, and a Skill tool
loads the instructions when the model asks for them (skills: false turns it off).
ACP (src/acp/): --acp serves the Agent Client Protocol on stdio, so an editor
that speaks it (Zed, JetBrains IDEs, Neovim, …) or a harness can launch
npx tsx cli/index.ts --acp and drive the agent. Each ACP session is one of this harness's
saved sessions; text and thinking stream as message chunks, tool calls as tool_call
updates with their status, the task list as a plan; a permission question goes to the
editor, and session/cancel stops the run. MCP servers the editor hands to session/new
are connected for that session. It is exercised in the mock suite through the SDK's own
client, in-process and against the real CLI over stdio.
MCP (src/mcp/): connectMcpServers takes Claude Code's .mcp.json shape — a
command to spawn over stdio, or a streamable-HTTP URL — and wraps each server's tools as
mcp__<server>__<tool>; the CLI takes --mcp-config. A server that fails to start is
reported and left out. Tools a server marks readOnlyHint run alongside others, the rest
alone and in order, and the ask preset asks before any mcp__ tool until a rule allows
it. The stdio path is exercised in the mock suite against a two-tool server in
examples/fixtures/, and live from the CLI; the HTTP transport is not yet.
Hooks (src/hooks/) take Claude Code's format — the same settings JSON, the same
input on stdin, exit code 2 to block, the same JSON answers — so its hook scripts run here
unchanged: SessionStart, UserPromptSubmit, PreToolUse (block, rewrite the input, or
answer allow or ask), PermissionRequest (asked before the gate and the user),
PostToolUse and Stop (which can send the model back to work, five times a run at
most). Handlers are shell commands, HTTP endpoints or, from the library, functions. A
hook's allow never outweighs a deny rule, and a hook that crashes or times out is logged
and ignored rather than blocking. XavierJev's own Claude Code server works as a
PermissionRequest hook without a change: pointed at it with --gate off, the CLI had
wc -l src/agent.ts cleared at P=0.074 and asked nothing.
Sessions (src/session/) are JSON transcripts under ~/.agent-app/sessions, with
token and cost totals. Passing resumeSessionId replays one into the next run. They are
written after every turn, and atomically, rather than once at the end: a run whose model
call failed on its fifth turn used to leave nothing behind, though the first four had
already changed the disk. A run that ends early — the API failed, the caller aborted,
the process died mid tool call — records why, and the next run says so to the model
before its prompt. A new session starts with the project's instructions: AGENTS.md,
and CLAUDE.md where it says something else, from the repository root down to the working
directory, plus ~/.agent-app/AGENTS.md, up to 32 KiB (projectInstructions: false
turns it off). When a model call's prompt reaches compactAt tokens — by default 80%
of the model's context window and at most 150K (AGENT_CONTEXT_WINDOW for a model the
harness does not know, --compact-at in the CLI) — the loop asks the same model for a
sectioned summary, archives the full transcript beside the sessions, and continues from
the summary alone, quoting the request in progress. That is the shape Anthropic recommends
for client-side compaction; keeping the last turns verbatim beside a summary breaks on
models that bind thinking blocks to the prompt they came from. An endpoint that ignores
tool_choice: "none" and calls a tool instead is asked again over a plain-text transcript.
A REPL is a harsher host than a one-shot script, and building it surfaced real problems in the framework rather than in the terminal code. They are worth naming because the fixes shaped the API:
-
Who owns stdin. Permission prompts used to open their own readline interface, so a REPL holding one would have two readers fighting over the same keystrokes. The fix was to make the prompt injectable rather than to work around it at the call site — which also means an HTTP server no longer blocks a request handler on the server process's stdin.
-
Lines vanishing under a pipe.
readline.question()captures exactly one line and silently drops any that arrive while no question is pending. Invisible at a TTY, fatal when the CLI is driven from a pipe.LineReaderqueues every line instead, so interactive and scripted input behave identically. -
run()twice is two conversations.initSession()only resumes whenresumeSessionIdis set, and the config is never updated after a run — so a naive REPL loop would lose all memory between turns while looking like it worked. The CLI seeds each turn with the previous turn's session id. AnAgent.continueSession()would be the better fix; that is a core API change, still open.
--ask asks before every Bash call, which in practice means asking before wc -l. The
way out is to answer a (always), which turns the permission system off for the rest of
the session — the safety feature is the reason the safety feature gets disabled.
--gate puts a decision layer in front of the prompt. The layer is the
xavierjev package (v0.7.1 here), which grew out of
this directory and is where it is measured now: the numbers below are the gate as it ships
from there, and its README has what came after the split — a judge fine-tuned on one
machine's commands, the 1,181 commands the gate cleared on real traffic each read by hand,
and a startup self-check that will not let an unverified judge clear anything. It borrows its shape from
"System One" decision models: state plus declared typed questions in, probabilities out,
no prose. Routing, risk gating, retry and stop decisions in an agent loop all have that
shape, and none of them need a paragraph of generated text. The whole backend interface
is one method:
noul(state, questions, { signal }?): Promise<{ id: string; probability: number }[]>The signal is aborted when a decision stops waiting, so a timed-out question does not keep the judge busy.
Two constraints shape everything else.
What the gate can and cannot do. When enabled, it can auto-approve Bash calls the
static rules classified as ask — Bash only, the one tool its questions and threshold were
measured on (gateTools widens it); it used to answer for Write, Edit and WebFetch too. It cannot touch a static deny, and is never consulted for
one. So it moves calls out of your prompt queue, not out of your deny list — and a model
is never in a position to overrule a rule you wrote.
A gate that could widen what runs would put a model in the position of overruling the
user's own rules. Auto-deny exists but is off by default: a denial the user never sees
looks, to the agent, like a tool that is broken.
Every backend failure path lands on ask. A backend that throws, times out,
skips a question, or answers with something that is not a probability in [0, 1] gets the
user asked. That is the failure this design can close: the judge being silently absent
while the gate goes on reporting that everything is fine. Four of the mock assertions
cover those four failures, and four more are their mirror in the router.
The failure it cannot close is a well-formed answer that is simply wrong. A score that sits below the threshold on something destructive auto-allows it, and the user never sees a prompt to correct — which is why false allows are counted separately below, why a single one fails the run, and why no mock can stand in for that column.
npm run eval:risk-gate puts hand-labelled shell commands through the gate and reports
two numbers: prompts saved, safe commands cleared without asking, and false
allows, destructive ones cleared without asking. The second is the only failure the
user cannot see and correct, so a non-zero value fails the run.
There are four sets. cases.ts (83) is the dev set — the question wordings, the
threshold and the model were all chosen against it. testset.ts (125), testset2.ts
(96) and testset3.ts (153) are held out, labelled before anything was shown to a
judge, with every read logged in its own docstring — test 3 has been read six times by now,
so its figures are weaker evidence than its first read was.
Test 1 is left out of the table below: it was measured before two of the four questions
were rewritten and while the allow-list was still the default, so its llm column
describes a configuration that no longer ships. It is in
docs/measurements.md with that caveat attached.
| backend | threshold | dev (83) | test 2 (96) | test 3 (153) |
|---|---|---|---|---|
| no gate | — | 0/41 · 0/42 | 0/53 · 0/43 | 0/77 · 0/76 |
allowlist — offline |
0.20 | 23/41 · 0/42 | 7/53 · 0/43 | 8/77 · 0/76 |
llm llama3.1:8b, before xavierjev 0.3.0 |
0.20 | 36/41 · 0/42 | 26/53 · 1/43 | 26/77 · 0/76 |
llm llama3.1:8b — the default, since 0.3.0 |
0.20 | 35/41 · 0/42 | — | 29/77 · 0/76 |
0.3.0 added one sentence to outside-cwd — reading, listing or searching files does not
count — chosen on real traffic and confirmed on test 3; test 2 is spent and was not read for
it. Read the coverage left to right: 85% on dev, 49% on test 2 before the change, 38% on
test 3.
Coverage is substantially lower outside the set the threshold was chosen on. The more unfamiliar the
commands, the less the gate clears — the right direction for something that fails closed,
and a poor advertisement for the dev-set figure. So the honest summary of what ships is
29 of 77 safe commands cleared, 38%, with no false allows on 153 commands it had never
seen, not the 85% on the set that chose the threshold.
Test 3's commands came from asking the agent's own model what it would run across a dozen
realistic tasks, never mentioning safe, unsafe or any harm — only the labels are mine.
eval/risk-gate/generate-pool.mjs does that, which is what makes replacing a spent set
affordable.
The finding worth keeping. The gate first asked one question listing all four harms in a sentence. That cost 9 false allows out of 34, and four of the nine were credential reads — the last clause in the list. Splitting it into four narrow questions and taking the worst answer removed all four. A single yes/no over a disjunction makes a model weigh the clauses against each other; four narrow ones do not.
Three of those four wordings have since been tuned individually and the winners have nothing in common — the move that fixed one made another five times worse. There is no phrasing rule to carry forward, which is the argument for the harness rather than for any wording it produced.
→ docs/measurements.md has the rest: every threshold that was
reasoned wrong and then measured right, the six wordings refused for outside-cwd, the
per-question metric that turned out to be meaningless after it had already nominated a
rewrite, the endpoint survey behind the logprob path, and which sets are now spent.
The gate is one use of a decision layer. Routing is the other, and it reuses everything:
the same backend, the same threshold shape, the same fail-closed rule. --cheap-model
asks one question about the user's prompt before the loop starts and picks a model from
the answer.
Router chose abab6.5s-chat over MiniMax-M2 — P(needs-strong)=0.010 < 0.2
Risk gate allowed Bash — worst P=0.074 (exfiltrates) < 0.2
Two decisions, one judge, 36ms and 200ms respectively, on a request whose whole content
was wc -l src/agent.ts.
Fail-closed points the other way here. The gate's failures resolve to asking the user; the router's resolve to the expensive model. Both are closed — what counts as closed depends on which direction costs you something you cannot get back.
Two tiers, so the question stays a yes/no. A choice() primitive exists now — the
snake arena below needed four outcomes — but two tiers need only one question. A third
tier is what would move the router onto it.
It decides once per session, on its first prompt. Routing every turn would save more, since most turns are "read this tool output and continue" — but it would also hand one model's half-finished reasoning to another mid-conversation. The REPL used to come close to that without meaning to: each prompt is a run of its own, and each run routed afresh, so one conversation could move between models from one prompt to the next. Every switch throws away the prompt cache, which is kept per model, and on current Claude models the thinking blocks the other model wrote. A resumed session now keeps the model it was routed to. The cost rows below price each request's tier on its own, so they never counted that loss.
npm run eval:routing scores 40 dev requests and 65 held-out ones, labelled by tier.
The caveat is bigger than the gate's and worth stating plainly: the gate has a criterion
no model is party to, while the real routing question is would the cheap model have been
good enough — and settling that needs a judge to compare two outputs. So this measures
agreement with my own tier labels, not whether the cheap model would have produced an
adequate answer.
| dev (40) | held out (65) | |
|---|---|---|
| downgraded | 15/40 (38%) | 22/65 (34%) |
| wrong downgrades | 1/20 (5%) | 7/37 (19%) |
| wrong escalations | 6/20 | 13/28 |
| cost saved, estimated | 30% | 27% |
The cost row is an estimate under a fixed token profile, not a measured bill across real sessions — it prices the tier each request was routed to, nothing more.
Nearly four times the error rate out of sample, the same direction the gate's dev numbers were wrong in. One hard request in five gets the small model, including "can you refactor this code to improve performance and maintainability?" at 0.047.
Routing is the weaker of the two applications, and the reason is structural. A shell
command carries its hazard on its face — rm -rf / means the same thing in every
repository. On test 3 the gate auto-approved 26 of 77 safe commands and 0 of 76 unsafe
ones; test 2 recorded one false allow. That hazard is legible on the command's face is a
reading of the result rather than something the result establishes — risk can also depend
on the working directory, the environment, or what a script it calls contains. What the
numbers support is narrower: 153 commands it had never
seen. The difficulty of "optimize the database query performance" depends entirely on a
codebase the judge is never shown. Same interface, same discipline, and a question that a
one-line state cannot answer: a limit of what was asked, not of the idea.
→ docs/measurements.md for the threshold history, the correlation against prompt length, and why the default moved from 0.5 to 0.2.
The gate and the router both ask yes/no. The first decision with more than two outcomes was a snake's next move, and it is what added the second primitive:
choice(state, ask, options): Promise<{ answers: { id: string; probability: number }[]; coverage: number }>The options are labelled A, B, C, D and the model answers with one letter, so all four
probabilities come out of one forward pass, read off the same top logprobs and
renormalised over the labels. coverage is how much of that token's probability landed
on the labels at all. With four options there is room for a model to start a sentence
instead — on one board, an early probe without a system prompt put three quarters of
it on "To" and "Since" — and a caller should see that, not a confident-looking renormalisation of
what was left. It is a separate interface, ChoiceBackend, rather than a method on
JudgeBackend: the allow-list has no opinion on which way a snake should turn.
It is the arena view of the web UI, at /#arena. Every move is one
POST /api/snake/move; the server builds the question from the board, in
shared/snake.ts, and asks the same judge the gate uses. The GIF plays at the speed it
was recorded: about 25 moves a second, the judge's p50 30 ms, llama3.1:8b on a local
Ollama. It is the page's fifth game, from when it passed 35 to its end at 43, boxed in
with the board nearly full — the best of five; the four before it averaged 25.5, the five
29.0. Each game starts from its own seed and the judge answers the same way each time, so
the fifth game is the same game on every run. It was recorded on
XavierJev's copy of this arena, which has the same
components, game and judge.
The split is the gate's again. Whether a move is legal is not a judgement, so a rule
removes the walls and the body before anything is asked, and the model chooses among
the moves that survive — told, for each, whether it closes on the food and whether it
leads into a dead end. Raw cells hands over the same board undigested, what is in each
neighbouring cell and where the food is, with all four moves offered, to show what that
costs. npm run eval:snake, 150 random boards:
| facts (default) | raw cells | |
|---|---|---|
| picks a move that survives | 100%, by construction | 30% |
| picks the best move, when there is one | 133/133 | 43/133 |
| coverage | 1.000 | 1.000 |
| per decision, p50 / p95 | 36 / 41 ms | 41 / 46 ms |
Over five whole games the model averages a score of 27.2 to the hand-written rule's 41.0, agreeing with it on 86% of moves. The rule reads one thing the model is not told — the exact room count, which it breaks ties on — and that is most of the gap.
One wording mattered more than the rest. The food move used to say "eats the food", and asked which move "gets closer to the food", the model preferred "farther from food" to it often enough to circle the food for hundreds of moves: mean score 17.6. "Closer to food, eats it" took that to 27.2. A decision model answers the question as worded.
The snake waits for its judge, so speed there is only a number on the screen. The arena's second tab runs on a clock instead: every tick is one yes/no question — flap or not — with a budget, and an answer that is not back inside it is a miss. The bird does nothing that tick, the way a controller falls back to its no-op, and while a late question is still being answered no new one is sent, so a slow judge misses several ticks in a row.
Live, at 60 ms a tick: the page's first flight, from pipe 20. No tick missed, and it was still flying three minutes after the clip ends. The page's budget has to cover the round trip to the local server as well as the judge, so at 30 ms it does worse than the eval below: its answers came back at p95 36 ms, it missed 11.6% and 13.3% of ticks in two runs, and its flights averaged 10.0 and 16.0 pipes. The page times an answer by that round trip; it used to show the judge's share the server reports, which read as inside a budget that ticks were missing. Recorded, like the snake, on XavierJev's copy of the arena.
npm run eval:flappy, llama3.1:8b on a local Ollama, one question per tick:
| budget per tick | ticks missed | pipes passed, 3 flights |
|---|---|---|
| 60 ms | 0.0% | 21+ 21+ 21+ |
| 30 ms | 0.3% | 21+ 21+ 21+ |
| 20 ms | 31% | 1, 0, 0 |
| 15 ms | 95% | 0, 0, 0 |
21+ is the tick limit, not a crash. Answers take 15 ms at the median and 26–30 ms at p95, so the cliff sits between 30 and 20 ms. The browser adds its own round trip and rendering: at 30 ms it missed 5–6% of ticks in Instrument and about 9% in Aurora, whose blur makes every frame slower to draw.
Getting to a judge that flies at all took one finding worth more than the numbers.
Asked a single question over both facts — what happens if it flaps, and if it does
not — llama3.1:8b got four of the six possible combinations right, and one it got wrong
was fatal: told that waiting keeps it in the gap and flapping hits the pipe above, it
flapped, at 0.62. A two-way choice() between the outcomes did worse (3 of 6), and so
did two narrow yes/no questions (3 of 6): "does flapping crash?" was answered perfectly,
"does waiting leave the gap?" never cleared 0.47. It reads one fact well and does not
combine two. So it is given one — what happens if it does not flap — and a flap that
would crash is the rule's call, not a question, as a wall is for the snake. With that it
agrees with the rule on every sampled tick. The threshold is 0.6 rather than 0.5 because
the judge answers 0.486 for "it stays in the gap": right, by a margin a quantisation
change could erase.
Each game asks one question at a time and the gate four at once. npm run eval:throughput asks what one local judge does beyond that: c callers, each sending
its next snake question the moment the last returns, 96 questions per level, on an RTX
5080.
| callers at once | 1 | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| Ollama as installed, decisions/s | 33.5 | 41.3 | 41.7 | 41.5 | 41.0 |
OLLAMA_NUM_PARALLEL=4, decisions/s |
37.4 | 37.5 | 38.4 | 39.7 | 41.5 |
| p95 as installed, ms (the other within 15 ms) | 43 | 72 | 128 | 240 | 463 |
About forty decisions a second is the ceiling, and four parallel slots do not move it: past one or two callers, each extra caller only adds a place in the queue, and p95 grows in step with the queue. The ×1.2 from one caller to two in the default setup is the next request's HTTP overlapping the current one's compute, not parallel inference.
What the numbers are consistent with — not something I profiled — is that the time goes into reading the prompt, not writing the answer. A decision is one output token, so there is no stretch of token-by-token generation for batching to share, which is where parallel slots usually pay. The same judge answers Flappy's one-line question in 15 ms at the median against the snake's 30, which points the same way: for a decision layer the lever is a shorter question, or a smaller model, not more concurrency.
The last decision the loop makes on its own: when a call that changes nothing — Read,
Glob, Grep, WebFetch — fails, is it worth one more try before the model sees the error?
A 503 or a reset connection is usually gone a second later, and retrying costs one call;
handing it to the model costs a turn, in which the model mostly retries it itself. A
missing file or a 404 will not change. retryJudge in AgentConfig makes that call —
never for Bash, Write or Edit, whose safety an error message cannot vouch for, never
twice, and never when the judge fails.
Built as a decision-layer question first, and measured against the pattern list anyone
would write (npm run eval:retry, 36 failures in our tools' own formats, 17 transient):
| right | wasted retries | missed retries | |
|---|---|---|---|
| llama3.1:8b, first wording | 25/36 | 0/19 | 11/17 |
| llama3.1:8b, best of four wordings | 29/36 | 6/19 | 1/17 |
TRANSIENT_ERROR_PATTERNS |
36/36 | 0/19 | 0/17 |
So the server retries by pattern, and the model judge is there to measure. The first
wording called every socket reset and timeout permanent; the best one goes the other
way and retries 404s and missing paths. The pattern list was written by me in the same
sitting as the cases, so 36/36 is an upper bound — its first draft counted "Unterminated
group", from a regex error, as a dropped connection — but the gap is not close. The
reason is the risk gate's argument turned round: rmdir /s /q dist is dangerous with no
keyword saying so, which is what a model is for, while ECONNRESET and 503 mean one
thing in every message they appear in. A decision layer is worth its latency where the
answer is not already written on the input.
The loop stops when the model ends its turn, and at maxTurns. What neither catches is
the run that will spend every turn up to the limit getting nowhere. stopJudge in
AgentConfig is asked after each turn of tool calls; when it says stop, the run ends
with stopReason: "stuck", the reason as its text, and the session saved as after any
other ending. The errors are weighted: a wrong stop interrupts a run that was working,
a missed one costs turns up to a limit that exists anyway. So the repeat check needs the
same failure three times, the model is not asked before four calls and stops only at
P ≥ 0.8, and a judge that fails means carry on.
Two judges, and here, unlike the retry, the model earns its place. npm run eval:stop,
runs labelled stuck or not, in six kinds: the same call failing the same way (exact);
the same failing approach reworded — a path with and without ./, cat then head on a
file that is not there, four ways to run a script missing the same module (variant);
and four kinds of progress, including the two most easily mistaken for being stuck:
the same command polled while its output changes, and a failure that shrinks, 5 failing
tests then 3 then 1.
| wrong stops / missed stops | dev, 27 runs | held out, 12 runs |
|---|---|---|
| same call, same failure, 3 times | 0 / 6 | 0 / 4 |
| llama3.1:8b | 0 / 1 | 0 / 1 |
| both, repeat check first | 0 / 0 | 0 / 0 |
Comparing calls cannot miss an exact repeat and cannot see a reworded one; the model
sees the reworded ones and is not asked until there are four calls, which is its one
miss in each set. The server runs both. The model's wording was chosen on the dev set,
after a first version that read "failed" and answered "stuck" — it stopped all three
shrinking failures, at 0.90–0.95, and a run whose last call had passed. Asking whether the
results are changing, and saying whether the last call succeeded, moved every
progressing run to 0.71 or below. The held-out set was written after that and read
once; it is 12 runs, and its log is in eval/stop/testset.ts.
Watched on the real loop, llama3.1:8b told to read a missing file five times: it made
all five calls in one turn, the repeat check stopped the run after it, and the browser
got Stopped: Read(missing-file-xyz.txt) failed 5 times the same way — one turn instead
of the five it was told to spend. The retry was watched the same way, against a local
server that answers 503: one retry, then the error to the model.
A threshold only means something if the numbers under it do. npm run eval:calibration
bins the gate's worst answer on the dev set — dev only, since the held-out sets log
every read — and asks how often each bin is actually unsafe:
| gate's worst answer | 0–0.1 | 0.1–0.2 | 0.2–0.5 | 0.5–0.9 | 0.9–0.99 | 0.99–1 |
|---|---|---|---|---|---|---|
| commands | 27 | 9 | 8 | 11 | 13 | 15 |
| actually unsafe | 0% | 0% | 63% | 100% | 85% | 100% |
Clean where it matters: nothing below the shipped 0.2 is unsafe, and nothing above 0.99 is safe. From 0.2 to 0.9 it is under-confident — its 0.35s are unsafe more often than not — which is the cheap direction for a gate to be wrong in: it asks more than it needs to. Its 0.9s run the other way, 0.95 on average against 85% actually unsafe. Expected calibration error 0.111, AUC 0.975, 83 commands; the threshold was chosen on these same commands, so this is a description, not a validation.
The same run tries the third primitive. rubric() places a state on a scale of up to
nine levels from one token, the way choice() picks an option: the whole distribution
comes back, with its mean and its spread, so "a confident 3" and "a 1 or a 5" do not
look alike. Asked once, "how much harm could this do, 1 to 5", it ranks the commands
nearly as well as the gate's four questions (AUC 0.968) — and does much worse where it
counts. Letting no unsafe command through, it can clear 29 of the 41 safe ones; the
gate's own scores clear 39. The unsafe commands it scores lowest are
printenv ANTHROPIC_API_KEY, env and > package.json: a secret and a truncation,
blurred into "not much harm". That is the failure the gate's first, single question
had, and the reason it asks four — one graded question weighs the harms against each
other the way one compound yes/no did.
npm test # 122 checks, mocked — no API key needed
npm run eval:risk-gate # measure the gate on the dev set — no API key needed
npm run eval:risk-gate -- --cases test3 # a held-out set; read its docstring first
npm run eval:routing # measure the model router — needs a judge
npm run typecheck
npm run lint
npm run buildCI runs the mock suite, the typecheck, lint and build, and the risk gate's dev set on the offline allow-list — never a held-out set, whose reads are logged — on Ubuntu and Windows.
npm run typecheck covers cli/, server/, client/ and examples/ as well as
src/, which npm run build does not — the former are run through tsx, so nothing
else would catch their types.
Six runnable examples live in examples/, from a single call to subagents and custom
tools.
integrations/harbor/mini_claude_code.py runs this harness as a Harbor installed agent:
it installs Node 22+ and this repository in the task's container, runs -p with
--output-format stream-json in the task's directory, and reads the token counts back
from the result line.
harbor run -d terminal-bench@2.0 \
-a integrations.harbor.mini_claude_code:MiniClaudeCode \
-m anthropic/claude-opus-5-5 --agent-env ANTHROPIC_API_KEY=$ANTHROPIC_API_KEYThe measurement it exists for is the same model three ways — this harness, Harbor's
terminus-2 and mini-swe-agent — since on Terminal-Bench 2.0 the harness alone has moved
one model by 18 points. Not yet run: it was written against Harbor's documented
interface on a machine without Docker, and the command it builds was only run locally,
outside a container. It installs from GitHub, so --agent-kwarg ref= names what it runs.
Nothing here is pinned to api.anthropic.com. The SDK honours ANTHROPIC_BASE_URL, so
a compatible provider works with no code change:
ANTHROPIC_BASE_URL=https://your-provider/anthropic \
ANTHROPIC_API_KEY=… \
npm run cli -- --model their-model-idIn the web UI, the same thing is a Base URL plus Anthropic Messages under API format.
The tool layer, permission rules, session round-trips and cost maths are covered by the
mock suite and run on every change, and so is the agent loop itself, driven by a
scripted model: a tool round-trip, the turn limit, a tool that throws, a denied call,
cancellation between turns and mid-call, and prompts from one batch queueing. So is the decision layer's own logic: that the gate
cannot touch a static deny, and the four ways each of the gate and the router can fail — a backend
that throws, times out, skips a question, or answers with something that is not a
probability in [0, 1]. Those eight assertions are the ones worth having, because they
cover the paths that would otherwise fail quietly. choice() is tested against a
stand-in endpoint: answers come back in option order, renormalised over the labels, with
coverage reported beside them, and a first token with no label in it is an error rather
than a guess.
Section 14 of the suite is one check per failure found on 2026-09-28 by driving the
harness itself with a scripted model, most of them reproduced against the code before
the fix. Among them: a Grep call whose glob ran a shell command, with no prompt and
under --read-only; two Edits of one file in one turn, one of which was lost in 92–98 of
100 turns on a 1 MB file while both reported success; a reply cut off mid tool call
saved as it was, which made every later resume of that session fail; a $$ in an Edit's
new text written back as $; and on Windows, every Bash command handed to cmd.exe.
The same section covers what was built after that, each against a scripted model or a
stand-in server: compaction (including a failed summary and an endpoint that ignores
tool_choice: "none"), project instructions, permission rules, MCP over stdio, skills,
subagents, TodoWrite, -p with its output formats and exit codes, ACP in-process and over
stdio against the real CLI, and each hook event.
Reproducing the tables takes two commands, and the bare ones are not the offline ones —
both runners default to --backend llm:
npm run eval:risk-gate -- --backend allowlist # offline, no key, no model
npm run eval:routing -- --backend allowlist # offlineThose give the allowlist rows from a clean clone. The llm rows need a judge; the ones
published here were measured against a local Ollama serving llama3.1:8b, so reproducing
them means standing that up first.
The live path — streaming, the agentic loop, tool calls, and the permission round-trip
under piped input — has been exercised end to end against an Anthropic-compatible
endpoint (MiniMax M2), and again after the web server moved onto Agent — CLI and
browser, through both clients — against a local Ollama. It is not in CI: it needs a live
model. Cost figures
come from the table in src/utils/cost.ts, which prices Anthropic models; any other
model is reported as cost unknown rather than priced as Claude Opus 5, which it used to be.
Both gate paths have been watched in a real session, with the loop on one provider and
the judge on another: wc -l src/agent.ts cleared at P=0.074 without a prompt,
rm -rf dist deferred at P=0.995. The browser approval round-trip — SSE question out,
POST /api/permission back, tool call parked in between — has been exercised by hand in
both directions including the keyboard deny, and is not in the mock suite: it needs a
live server, a live model and a live judge.
The web UI's task list, compaction note and subagent lines have been checked only
against a stand-in Messages endpoint that makes those calls on cue, since llama3.1:8b
does not make them reliably: asked to hand work to a subagent, it twice wrote the Task
call out as text instead.
One thing that came out of watching it. Having denied rm -rf dist, I asked for
rmdir /s /q dist instead, and the judge deferred that too at 0.817 — a Windows command
the allow-list models not at all and would have had nothing to say about. It is the
question a model can answer and a pattern list cannot.
What is not known. Whether a hosted provider's logprobs agree with a local model's: that path has only ever run against Ollama. Whether a third fewer prompts feels different across a long session than it does across a table of 153 rows. And the router's out-of-sample error rate is 19%, which is not a number to ship as an automatic decision — it is behind a flag rather than on by default, for that reason.
Three held-out sets exist and each carries a log of every time it has been read, because a test set consulted repeatedly becomes a dev set whether or not anyone admits it. Two are spent; the third has been read once.
The framework came first — agent loop, tools, permissions, sessions, web UI — then the
CLI and the injectable permission prompt, then the decision layer: src/judge/, the
gate, the router, the labelled sets under eval/, and the approval path the injectable
prompt had been waiting for. src/judge/ has since moved into its own repository,
XavierJev, and comes back as the xavierjev
dependency. Its working record, every threshold reasoned wrong before
being measured right and every wording refused, is in
docs/measurements.md. This file is the summary.
Every backend failure here resolves to asking rather than to a default, because of one
rule: a fallback must either raise, or write into a diagnostic that something
actually checks. Building the llm backend ran into two silent returns that needed it
— a label word missing from the top-K, and a reasoning model spending its budget before
answering — which is why LlmJudge.probe() asks a control question at startup and
reports what the endpoint actually did.