From 07a141b924e6389334fae991c0629f39c124ca8c Mon Sep 17 00:00:00 2001 From: Robert Keus Date: Wed, 12 Aug 2026 21:37:51 +0200 Subject: [PATCH] Document measured token efficiency --- README.md | 48 ++++++++++++++------- docs/token-efficiency.md | 93 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 126 insertions(+), 15 deletions(-) create mode 100644 docs/token-efficiency.md diff --git a/README.md b/README.md index 73be3f6..44ed929 100644 --- a/README.md +++ b/README.md @@ -83,12 +83,23 @@ that needs to exist, say the minimum about it — and [ESON](https://github.com/Green-PT/honey-eson), the wire format every subagent return comes back in. -**Why it cuts tokens.** Output is the bill. Builders write only the code the -spec demands (stdlib before custom, nothing speculative) and skip the -narration; auditors return verdicts, not essays. Fewer tokens per unit of -shipped work — not fewer gates. An epic that runs tests, an OWASP pass and a -WCAG pass still costs more than a one-shot that skips them and ships a 500; -what's gone is the waste, not the rigor. +**What is measured.** Honey's pinned, paired 23-task benchmark reduced output +by 29% (`p=.020`) and code by 43% (`p<.001`) on Claude Opus 4.8; on GPT-5.5, +output fell 20% (`p=.004`) and code 18% (`p<.001`). ESON's deterministic +five-document handoff benchmark used 3,151 o200k tokens versus 4,395 for +compact JSON, a 28% reduction after lossless round-trip checks. Total Honey +cost was a statistical tie on both providers because its prompt adds input and +caching differed. Peter has no control run without Honey and ESON, so these are +upstream component results, not a claim that Peter's $27.63 proof run would +have cost a specific amount otherwise. See the +[reproducible evidence, exact revisions, and limits](docs/token-efficiency.md). + +**Why use them.** Builders write only the code the spec demands (stdlib before +custom, nothing speculative) and skip narration; auditors return verdicts, not +essays. Fewer output tokens and lines per unit of shipped work — not fewer +gates. An epic that runs tests, an OWASP pass and a WCAG pass can still cost +more than a one-shot that skips them and ships a 500; what's targeted is waste, +not rigor. **Why runs have more context.** Every subagent return lands in the parent's context window and stays there for the rest of the drain. A narrated diff @@ -107,12 +118,14 @@ src/checkout/api.ts +stripe intent endpoint src/checkout/api.test.ts +4 cases ``` -That's a whole task return. Cheaper than JSON on the wire — no braces or -quotes per row — and self-checking: `[2]` declares the row count, so a +That's a whole task return. On record-heavy, cached handoffs, ESON removes +repeated keys and JSON punctuation; `[2]` also declares the row count, so a truncated return is detected and re-requested instead of silently losing -findings. One carve-out is absolute: anything touching auth, money, -migrations, deletes, or data loss keeps its full text. Honey compresses -everything except the things that hurt when compressed. +findings. It is not a universal win: its primer never amortizes without prompt +caching and scalar-only messages can be larger than compact JSON. One carve-out +is absolute: anything touching auth, money, migrations, deletes, or data loss +keeps its full text. Honey compresses everything except the things that hurt +when compressed. ESON is the message format only — `graph.jsonl` stays JSONL. @@ -183,10 +196,15 @@ tests/install.sh installer integration coverage **Why "peter"?** Named for [Peter Steinberger](https://x.com/steipete), whose July 2026 -question — "Are we still talking loops or did we shift to graphs yet?" — -sparked the graph-engineering framing this repo implements: a stable org graph -of specialist roles, a per-epic work graph of dependency-ordered tasks. No -affiliation or endorsement — just credit for the frame. +[question](https://x.com/steipete/status/2078277297791189132) — "Are we still +talking loops or did we shift to graphs yet?" — supplied no definition. Peter +is one concrete interpretation, informed by writeups from +[Carlos E. Perez](https://x.com/IntuitMachine/article/2078419526354378975), +[Opinion AI](https://emergingai.substack.com/p/graph-engineering-the-next-step-after), +and [AI Builder Club](https://www.aibuilderclub.com/blog/graph-engineering-guide-2026): +a stable role graph of specialists plus a persistent work graph of +dependency-ordered tasks. No affiliation or endorsement — just credit for the +prompt and the surrounding discussion. **Is it a framework?** No. A skill, four agent files, and a JSONL contract. Claude Code is the diff --git a/docs/token-efficiency.md b/docs/token-efficiency.md new file mode 100644 index 0000000..acce0e8 --- /dev/null +++ b/docs/token-efficiency.md @@ -0,0 +1,93 @@ +# Token-efficiency evidence + +Peter uses two upstream components to reduce output volume: + +- Honey constrains builders and auditors to the minimum code and prose needed. +- ESON encodes structured agent returns with record keys declared once. + +The evidence supports lower output tokens and fewer lines of generated code. +It does **not** establish a measured end-to-end cost saving for Peter: the +public Peter run has no equivalent no-Honey/no-ESON control arm. + +## ESON format benchmark + +Recomputed on 2026-08-12 from +[`Green-PT/honey-eson@d6809a1`](https://github.com/Green-PT/honey-eson/tree/d6809a131067e84faad6cc1ff47026664dcca988). +All five documents round-tripped losslessly before token measurement; the JS +and Python suites also passed (29 tests total). + +| Format | o200k tokens | vs compact JSON | Claude tokenizer estimate | vs compact JSON | +|--------|-------------:|----------------:|--------------------------:|----------------:| +| Compact JSON | 4,395 | baseline | 4,536 | baseline | +| Pretty JSON | 6,816 | +55% | 6,702 | +48% | +| Columnar JSON | 3,440 | -22% | 3,451 | -24% | +| ESON | 3,151 | -28% | 3,361 | -26% | + +The corpus contains a small review, large review, scalar envelope, nested +context, and tool results. ESON was 30% smaller than compact JSON on the large +review and 28% smaller on tool results, but **8% larger** on the scalar +envelope. + +The ESON primer measured 125 o200k tokens, versus 50 for columnar JSON. Its +extra 75 tokens break even after about two average record-heavy messages when +the primer is cached; without prompt caching, the benchmark says it never +breaks even. This is why Peter reserves ESON for repeated agent handoffs and +keeps `graph.jsonl` as JSONL. + +Reproduce: + +```bash +git clone https://github.com/Green-PT/honey-eson.git +cd honey-eson +git checkout d6809a131067e84faad6cc1ff47026664dcca988 +npm ci --ignore-scripts +npm test +npm run bench:formats +npm run bench:primer +``` + +The o200k count uses `gpt-tokenizer@3.4.0`. The Claude column uses +`@anthropic-ai/tokenizer@0.0.4`, which is a legacy estimate rather than an +exact count for current Claude models. + +## Honey paired benchmark + +Recomputed from the committed records in +[`Green-PT/honey-for-devs@b39339e`](https://github.com/Green-PT/honey-for-devs/tree/b39339e32e63721835756188d7ba08947ac7f709/bench). +Each result is the paired per-task median over 23 author-written tasks and three +runs. Continuous endpoints use a two-sided Wilcoxon signed-rank test; judge +scores use an exact sign test. + +| Model | Output delta | LOC delta | Total-cost delta | Tests, baseline → Honey | Judge W/L/T | +|-------|-------------:|----------:|-----------------:|------------------------:|------------:| +| Claude Opus 4.8 | -29%, `p=.020` | -43%, `p<.001` | -21%, `p=.104` (not significant) | 97% → 100% | 8/11/2, `p=.648` (tie) | +| GPT-5.5 | -20%, `p=.004` | -18%, `p<.001` | +14%, `p=.820` (not significant) | 100% → 99% | 6/8/7, `p=.791` (tie) | + +The significant result is less output and less code on both model families. +Quality was a judge tie, not a gain. Total cost was inconclusive and moved in +opposite directions: Claude reused the skill prompt through caching, while the +GPT run reported no cache reads and paid 573% more fresh/cache-creation input +(`p<.001`). Therefore Peter claims output reduction, not a proven dollar saving. + +Reproduce without API spend from the committed result records: + +```bash +git clone https://github.com/Green-PT/honey-for-devs.git +cd honey-for-devs +git checkout b39339e32e63721835756188d7ba08947ac7f709 +cd bench +node src/report.js --stamp full-opus48 +node src/report.js --stamp full-gpt55 +``` + +## Limits + +- The Honey tasks were written by Honey's author and are not an independent + external suite; 23 tasks are enough to observe an effect, not a leaderboard. +- Judge scores are noisy. Objective tests are the stronger correctness signal. +- ESON's format benchmark measures serialization, not model quality. Its + separate upstream comprehension suite is not reproduced here. +- Neither benchmark recreates Peter's full graph, retries, audits, or context + history. Those can dominate the final bill. +- No saving is claimed for auth, money, migrations, deletes, or other + irreversible payloads; Peter keeps those explicit rather than compact.