llmcheck

Four command line tools for keeping an LLM application honest.

• regress runs a suite of prompts against recorded baselines and exits non-zero when today's output has drifted
• mutate switches each of your guardrails off one at a time, re-runs your tests, and tells you which guardrails no test is covering
• costs aggregates a recorded call log into spend and latency per route
• report turns all three into one markdown file a non-technical client can read

Everything runs offline against a small built-in demo application with a fake model.

That's the one I care about. pii_redaction survived being switched off, which means no test in the suite would have noticed if it were deleted.

I built this after finishing two other agent projects. Both times I ended up doing the same three checks by hand, badly, at the end, and the second time I found a guardrail in the first project that had been sitting there untested for weeks. Nothing was wrong with it. It worked. But I'd have had no way of knowing if somebody had deleted it, and neither would the test suite, and that is a bad position to be in when the guardrail is the thing standing between a language model and a customer's refund. So instead of writing the check a third time I wrote it down.

Try it

git clone https://github.com/omarmaali/llmcheck
cd llmcheck
python -m venv .venv
.\.venv\Scripts\Activate.ps1        # bash: source .venv/bin/activate
pip install -e ".[dev]"

llmcheck regress src/llmcheck/demo/suites/demo_regress.yaml
llmcheck mutate  src/llmcheck/demo/suites/demo_mutate.yaml
.venv\Scripts\python -m llmcheck.demo.traffic --rounds 10
llmcheck costs report
llmcheck report

The demo app has three flaws in it on purpose, one for each tool to find. They stay there. regress passes 7/7 by default because its flaw is a second version of a prompt, and you have to point the suite at it.

regress

Cases live in YAML. Each one has an input, some checks, and optionally a golden file holding the output you last approved.

cases:
  - id: refund_above_the_cap
    input: {route: refund, ticket_id: ACME-1042, requested_usd: 1200}
    checks:
      - {type: json_path, path: "amount_usd", equals: 500.0}
      - {type: json_path, path: "message", not_contains: "1200"}
      - {type: golden, mode: exact}

Checks available: exact, contains, not_contains, matches, not_matches, json_path, and golden in exact, contains or similarity mode. The target is either a Python callable or a shell command that reads JSON on stdin and writes JSON on stdout.

The demo's first flaw is a second version of the summarise prompt that drops the ticket ID. Pointing the same suite at it:

$ llmcheck regress src/llmcheck/demo/suites/demo_regress.yaml \
    --set-env LLMCHECK_DEMO_PROMPT_VERSION=v2
suite demo-routes: 5/7 cases passed
  FAIL summarize_billing_ticket
      ! contains: missing 'ACME-1042'
      ! golden:similarity: similarity 0.823 vs threshold 0.90
  FAIL summarize_access_ticket
      ! golden:similarity: similarity 0.827 vs threshold 0.90
report written to examples/regress_v2_fail.json

That artifact is committed, so you can check those two numbers without running anything.

Goldens are only written when you ask for them with --update-goldens, and only for cases whose other checks passed.

Exit codes: 0 clean, 1 at least one case failed, 2 the suite file or the arguments are wrong, 3 a case wants a golden that doesn't exist yet.

mutate

This one came out of auditing my own earlier projects. I had six guardrails and a README claiming each one had a test. Deleting a guardrail and re-running the suite is the only way to find out whether that's true, and doing it by hand six times is dull enough that you do it once.

The plan names each guardrail and how to switch it off. Three mechanisms: an environment variable, a key in a config file, or a find-and-replace in a source file.

test_command: "python -m pytest tests/demo -q"
guardrails:
  - id: pii_redaction
    disable: {type: config, file: src/llmcheck/demo/demo_config.json,
              key: guardrails.pii_redaction, value: false}

Running it against the demo:

$ llmcheck mutate src/llmcheck/demo/suites/demo_mutate.yaml
baseline: python -m pytest tests/demo -q -p no:cacheprovider
disabling refund_cap ...
  killed    refund_cap
disabling pii_redaction ...
  survived  pii_redaction  <- not what the plan expected
disabling citation_required ...
  killed    citation_required
2 killed, 1 survived, 0 errored, score 67%
  pii_redaction survived: the suite still passes with this guardrail off
report written to examples/mutate_shipped.json

pii_redaction survives because the demo's tests check that the support route answers correctly, and none of them checks that the customer's email address came back redacted. You could delete the guardrail and the suite would stay green.

tests/demo_fixed/ holds the two tests that were missing. Add them and run the same plan:

$ llmcheck mutate src/llmcheck/demo/suites/demo_mutate.yaml \
    --test-command "python -m pytest tests/demo tests/demo_fixed -q -p no:cacheprovider"
  killed    refund_cap
  killed    pii_redaction
  killed    citation_required
3 killed, 0 survived, 0 errored, score 100%

Both artifacts are committed: examples/mutate_shipped.json and examples/mutate_with_missing_test_added.json.

Since the tool edits files and shells out to your test command, the safety rules:

• it refuses to run against a dirty working tree, or outside a git repository, unless you pass --allow-dirty or set require_clean_worktree: false in the plan. Regenerating the committed artifacts above tripped this on me
• it runs your test suite once before mutating anything and stops if it's already failing
• it snapshots the target file's bytes and its SHA-256 before writing, restores in a finally, and re-reads the file afterwards to confirm the hash matches. A SIGINT mid-run restores and exits 130
• a find-and-replace that matches nothing is a hard error
• on a timeout it kills the process group, so a test runner that started a server doesn't outlive the run
• exit code 1 from your test command means killed. Anything else, including pytest's "no tests collected", is reported as an error and the report says which code it got

llmcheck mutate exits 1 if any guardrail came back differently from what the plan declared, which is why the demo above exits 1.

costs

A thin wrapper round an OpenAI-compatible client that writes one JSON line per call. costs report reads the log back.

with CallRecorder(".llmcheck/costs.jsonl") as recorder:
    client = recorder.wrap(OpenAI())
    with recorder.route("support"):
        client.chat.completions.create(model="gpt-4o-mini", messages=[...])

Everything inside one route block shares a request ID, so a retry loop shows up as two calls against one request rather than disappearing into an average. That's the third flaw in the demo: the app retries once when the model returns nothing useful.

$ llmcheck costs report --log examples/costs.jsonl
route      model       calls  reqs  calls/req  tokens  cost     p50  p95
refund     demo-fast   10     10    1.0        485     $0.0001  6    6
summarize  demo-fast   17     10    1.7        1353    $0.0003  3    45
support    demo-smart  70     60    1.17       4770    $0.0301  18   25

97 calls over 80 requests, 1.21 per request
total $0.0305 using prices as of 2026-08-11

Two things fall out of that table. 17 of the 97 model calls, 17.5%, are retries nobody asked for: 10 in support and 7 in summarize, which at 1.7 calls per request is retrying most of the time. And support is 98.4% of the spend on 72% of the calls, because it's pointed at the expensive model.

The counts, tokens and dollars reproduce exactly. The latency columns won't. The fake model really does sleep, so p50 and p95 are measured on whichever machine ran it.

demo-fast and demo-smart are invented models with invented prices. See the note on pricing below.

report

$ llmcheck report --title "Demo app reliability report"

Reads whichever artifacts are present and writes one markdown file. Rendered example at examples/report.md.

The last section is fixed text listing what the report does not prove. It goes out with every report whether anyone wants it there or not.

How the pieces fit

flowchart LR
    A[your app] -->|recorder.wrap| L[(costs.jsonl)]
    S[suite.yaml] --> R[regress]
    A --> R
    P[plan.yaml] --> M[mutate]
    T[your test suite] --> M
    L --> C[costs report]
    R --> J1[regress.json]
    M --> J2[mutate.json]
    C --> J3[costs.json]
    J1 & J2 & J3 --> RP[report] --> MD[report.md]

Loading

Install

PyYAML is the only runtime dependency. pytest is the only development one, and openai is an optional extra you don't need for anything in this repo.

python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -e ".[dev]"
pytest -q

There's no make here.

Two Windows notes. PowerShell only reads something as a path if it starts with .\, so the leading .\ on the activate line is load bearing. And if python gives you a Microsoft Store advert instead of an interpreter, the venv isn't active in that window: either activate it, or call .venv\Scripts\python directly, which works either way.

Options

command	options
regress SUITE	--update-goldens, --set-env KEY=VALUE, --tag, --out, -v
mutate PLAN	--allow-dirty, --test-command, --only ID, --out
costs report	--log, --prices, --group-by, --out
report	--regress, --mutate, --costs, --title, --out

Suite keys: version, name, target (python or command, plus env and timeout_seconds), defaults.checks, goldens_dir, cases with id, input, checks, tags.

Plan keys: version, name, test_command, timeout_seconds, require_clean_worktree, working_dir, guardrails with id, description, disable, expect.

Environment: LLMCHECK_DIR sets where artifacts go, LLMCHECK_SKIP_DOTENV=1 ignores .env, LLMCHECK_TRACEBACKS=1 prints the traceback when a target raises.

What I've verified, and what I haven't

Verified on Python 3.13, running the commands in this file:

• 253 tests pass with no API key, no network and no .env. The suite sets LLMCHECK_SKIP_DOTENV=1, builds its fixtures under tmp_path, and clears the demo's environment switches. Running pytest with DEMO_DISABLE_REFUND_CAP=1 and LLMCHECK_DEMO_PROMPT_VERSION=v2 exported is itself one of the tests
• every number quoted above comes from a committed artifact under examples/, produced by the command shown next to it
• pip install -e . in a clean temporary directory, then the console script and python -m llmcheck. Also python -m build, checking the price table and the demo goldens are inside the wheel and not only in git
• PyYAML 6.0.3 publishes a cp313-cp313-win_amd64 wheel, checked against the PyPI JSON API before I picked it
• CI is green on ubuntu and windows, Python 3.13, and was green on its first run. It covers the test suite and the four demo commands. It does not run mutate against the demo, because that exits 1 by design
• the suite runs on Windows as well as Linux. It didn't at first. Running it on Windows found two bugs that Linux cannot reach, both now fixed and both with a test: a test_command holding a backslash path inside a double-quoted YAML scalar died with a raw scanner error, and the check that stops a config mutation from being a no-op compared bytes, so a CRLF file defeated it and a mutation that disabled nothing would have been scored

Now the other side of it.

The similarity check is lexical, not semantic. It compares overlapping four-character runs. Swap one word and it still scores 0.855. Say the same thing in different words and it scores 0.018:

"Refunds are available within 30 days of purchase." "You can get your money back up to a month after buying something."

Both are pinned in tests/test_similarity.py. A low similarity means go and look. I picked this over a sentence embedding model because embeddings need either a download or an API key, and that breaks the promise that a fresh clone works offline.

There's no price table for real providers, and that's on purpose. While building this I fetched OpenAI's pricing page twice and got two different sets of numbers for the same model, which was enough to convince me that a table I maintain by hand and forget to update is worse than a blank column. So pricing.json ships with the demo's fictional models only, and the report prints the date on whatever table you supply.

mutate proves less than it looks like it proves. A killed mutation means some test fails when that guardrail is off. It doesn't mean the guardrail is correct.

The patch mutator is textual. It's guarded against matching nothing, but it's a find-and-replace, not an AST transform, and it won't survive a refactor of the line it's aimed at. It edits files on disk too, so if your package is installed non-editable, the code your tests import isn't the code being patched.

Three of this repo's own tests run mutate against this repo, editing two tracked files and restoring them. Don't run two copies of the suite at once.

Token counts in the demo are characters over four. The real recorder reads whatever the provider puts in the usage field.

A config or env mutation proves nothing if the test builds the object itself. This is the one I'd want you to read. See the section below, where it cost me four false results on my own code.

Run against my own two projects

The demo above is a fixture I wrote, so it proves the tool works and nothing else. Here is what happened when I pointed it at the two agent projects I had already finished and already believed were well tested.

Ticket triage agent, nine guardrail rules. First pass scored 62%: five killed, three survived. I nearly stopped there. Checking the three survivors took about five minutes and all three were wrong. The tests do settings.max_steps = 4 and settings.kb_score_floor = 0.99 directly on the settings object, so switching the environment variable never reached them. Given handles that edit the check itself, all three were killed. The honest score is 9 of 9.

RAG support assistant, four guardrails. Answerability and the citation audit were killed. The two retrieval thresholds survived being loosened to 0.0 and were killed when tightened to 0.99, which is a lesson of its own: lowering a floor only tests anything if some input was sitting between the old value and the new one. relative_floor survived both directions and turned out to have a dedicated test that passes the value in as an argument, so no config change could reach it either.

That leaves one genuine gap, and I am keeping it. The system prompt in that project is guardrail three, six numbered rules telling the model to answer only from the retrieved context. The only test on it checks that the refusal string is embedded. I replaced "You answer ONLY from the numbered sources given in the CONTEXT block" with "Answer however you like" and the suite stayed green. The defence is that what a prompt says is validated by eval runs against a real model, and that project's README reports those numbers. It is not something pytest can check. But nothing in CI would notice if that prompt were gutted.

So: four false survivors across two projects, every one of them a properly tested guardrail. A number I would have published as 62% was really 100%. The tool now says so itself, in the CLI and in the report, whenever an env or config mutation survives. A patch survivor is evidence. A config survivor is a question.

Repository layout

path	what's in it
src/llmcheck/cli.py	argparse, subcommands, exit codes
src/llmcheck/config.py	the .env loader
src/llmcheck/similarity.py	character n-gram cosine
src/llmcheck/_util.py	atomic writes, percentiles, path display
src/llmcheck/regress/	suite parsing, the check types, the runner
src/llmcheck/mutate/	plan parsing, the mutators, the runner
src/llmcheck/costs/	the client wrapper, the price table, aggregation
src/llmcheck/report/	markdown assembly
src/llmcheck/demo/	the flawed demo app, its config, suites and goldens
tests/	253 tests
tests/demo/	the demo app's own tests, the target mutate runs against
tests/demo_fixed/	the two tests that were missing
examples/	committed artifacts backing every number in this file

Using it on your own project

regress needs a callable that takes a dict and returns something JSON serialisable, or a command that reads and writes JSON. mutate needs a way to switch each guardrail off from outside the code. If a guardrail can't be disabled by an environment variable or a config key, that's usually a sign it's woven into a function doing three other things.

costs needs the wrapper in place before it can tell you anything.

Licence

MIT.