Skip to content

Publish measured token-efficiency evidence #12

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
robertkeus merged 2 commits into from
Aug 12, 2026
Merged

Publish measured token-efficiency evidence #12

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
07a141b
Document measured token efficiency
robertkeus Aug 12, 2026
ef91c4d
Merge pull request #11 from robertkeus/codex/token-efficiency-proof
robertkeus Aug 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 33 additions & 15 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,12 +83,23 @@ that needs to exist, say the minimum about it — and
[ESON](https://github.com/Green-PT/honey-eson), the wire format every subagent
return comes back in.

**Why it cuts tokens.** Output is the bill. Builders write only the code the
spec demands (stdlib before custom, nothing speculative) and skip the
narration; auditors return verdicts, not essays. Fewer tokens per unit of
shipped work — not fewer gates. An epic that runs tests, an OWASP pass and a
WCAG pass still costs more than a one-shot that skips them and ships a 500;
what's gone is the waste, not the rigor.
**What is measured.** Honey's pinned, paired 23-task benchmark reduced output
by 29% (`p=.020`) and code by 43% (`p<.001`) on Claude Opus 4.8; on GPT-5.5,
output fell 20% (`p=.004`) and code 18% (`p<.001`). ESON's deterministic
five-document handoff benchmark used 3,151 o200k tokens versus 4,395 for
compact JSON, a 28% reduction after lossless round-trip checks. Total Honey
cost was a statistical tie on both providers because its prompt adds input and
caching differed. Peter has no control run without Honey and ESON, so these are
upstream component results, not a claim that Peter's $27.63 proof run would
have cost a specific amount otherwise. See the
[reproducible evidence, exact revisions, and limits](docs/token-efficiency.md).

**Why use them.** Builders write only the code the spec demands (stdlib before
custom, nothing speculative) and skip narration; auditors return verdicts, not
essays. Fewer output tokens and lines per unit of shipped work — not fewer
gates. An epic that runs tests, an OWASP pass and a WCAG pass can still cost
more than a one-shot that skips them and ships a 500; what's targeted is waste,
not rigor.

**Why runs have more context.** Every subagent return lands in the parent's
context window and stays there for the rest of the drain. A narrated diff
Expand All @@ -107,12 +118,14 @@ src/checkout/api.ts +stripe intent endpoint
src/checkout/api.test.ts +4 cases
```

That's a whole task return. Cheaper than JSON on the wire — no braces or
quotes per row — and self-checking: `[2]` declares the row count, so a
That's a whole task return. On record-heavy, cached handoffs, ESON removes
repeated keys and JSON punctuation; `[2]` also declares the row count, so a
truncated return is detected and re-requested instead of silently losing
findings. One carve-out is absolute: anything touching auth, money,
migrations, deletes, or data loss keeps its full text. Honey compresses
everything except the things that hurt when compressed.
findings. It is not a universal win: its primer never amortizes without prompt
caching and scalar-only messages can be larger than compact JSON. One carve-out
is absolute: anything touching auth, money, migrations, deletes, or data loss
keeps its full text. Honey compresses everything except the things that hurt
when compressed.

ESON is the message format only — `graph.jsonl` stays JSONL.

Expand Down Expand Up @@ -183,10 +196,15 @@ tests/install.sh installer integration coverage

**Why "peter"?**
Named for [Peter Steinberger](https://x.com/steipete), whose July 2026
question — "Are we still talking loops or did we shift to graphs yet?" —
sparked the graph-engineering framing this repo implements: a stable org graph
of specialist roles, a per-epic work graph of dependency-ordered tasks. No
affiliation or endorsement — just credit for the frame.
[question](https://x.com/steipete/status/2078277297791189132) — "Are we still
talking loops or did we shift to graphs yet?" — supplied no definition. Peter
is one concrete interpretation, informed by writeups from
[Carlos E. Perez](https://x.com/IntuitMachine/article/2078419526354378975),
[Opinion AI](https://emergingai.substack.com/p/graph-engineering-the-next-step-after),
and [AI Builder Club](https://www.aibuilderclub.com/blog/graph-engineering-guide-2026):
a stable role graph of specialists plus a persistent work graph of
dependency-ordered tasks. No affiliation or endorsement — just credit for the
prompt and the surrounding discussion.

**Is it a framework?**
No. A skill, four agent files, and a JSONL contract. Claude Code is the
Expand Down
93 changes: 93 additions & 0 deletions docs/token-efficiency.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
# Token-efficiency evidence

Peter uses two upstream components to reduce output volume:

- Honey constrains builders and auditors to the minimum code and prose needed.
- ESON encodes structured agent returns with record keys declared once.

The evidence supports lower output tokens and fewer lines of generated code.
It does **not** establish a measured end-to-end cost saving for Peter: the
public Peter run has no equivalent no-Honey/no-ESON control arm.

## ESON format benchmark

Recomputed on 2026-08-12 from
[`Green-PT/honey-eson@d6809a1`](https://github.com/Green-PT/honey-eson/tree/d6809a131067e84faad6cc1ff47026664dcca988).
All five documents round-tripped losslessly before token measurement; the JS
and Python suites also passed (29 tests total).

| Format | o200k tokens | vs compact JSON | Claude tokenizer estimate | vs compact JSON |
|--------|-------------:|----------------:|--------------------------:|----------------:|
| Compact JSON | 4,395 | baseline | 4,536 | baseline |
| Pretty JSON | 6,816 | +55% | 6,702 | +48% |
| Columnar JSON | 3,440 | -22% | 3,451 | -24% |
| ESON | 3,151 | -28% | 3,361 | -26% |

The corpus contains a small review, large review, scalar envelope, nested
context, and tool results. ESON was 30% smaller than compact JSON on the large
review and 28% smaller on tool results, but **8% larger** on the scalar
envelope.

The ESON primer measured 125 o200k tokens, versus 50 for columnar JSON. Its
extra 75 tokens break even after about two average record-heavy messages when
the primer is cached; without prompt caching, the benchmark says it never
breaks even. This is why Peter reserves ESON for repeated agent handoffs and
keeps `graph.jsonl` as JSONL.

Reproduce:

```bash
git clone https://github.com/Green-PT/honey-eson.git
cd honey-eson
git checkout d6809a131067e84faad6cc1ff47026664dcca988
npm ci --ignore-scripts
npm test
npm run bench:formats
npm run bench:primer
```

The o200k count uses `gpt-tokenizer@3.4.0`. The Claude column uses
`@anthropic-ai/tokenizer@0.0.4`, which is a legacy estimate rather than an
exact count for current Claude models.

## Honey paired benchmark

Recomputed from the committed records in
[`Green-PT/honey-for-devs@b39339e`](https://github.com/Green-PT/honey-for-devs/tree/b39339e32e63721835756188d7ba08947ac7f709/bench).
Each result is the paired per-task median over 23 author-written tasks and three
runs. Continuous endpoints use a two-sided Wilcoxon signed-rank test; judge
scores use an exact sign test.

| Model | Output delta | LOC delta | Total-cost delta | Tests, baseline → Honey | Judge W/L/T |
|-------|-------------:|----------:|-----------------:|------------------------:|------------:|
| Claude Opus 4.8 | -29%, `p=.020` | -43%, `p<.001` | -21%, `p=.104` (not significant) | 97% → 100% | 8/11/2, `p=.648` (tie) |
| GPT-5.5 | -20%, `p=.004` | -18%, `p<.001` | +14%, `p=.820` (not significant) | 100% → 99% | 6/8/7, `p=.791` (tie) |

The significant result is less output and less code on both model families.
Quality was a judge tie, not a gain. Total cost was inconclusive and moved in
opposite directions: Claude reused the skill prompt through caching, while the
GPT run reported no cache reads and paid 573% more fresh/cache-creation input
(`p<.001`). Therefore Peter claims output reduction, not a proven dollar saving.

Reproduce without API spend from the committed result records:

```bash
git clone https://github.com/Green-PT/honey-for-devs.git
cd honey-for-devs
git checkout b39339e32e63721835756188d7ba08947ac7f709
cd bench
node src/report.js --stamp full-opus48
node src/report.js --stamp full-gpt55
```

## Limits

- The Honey tasks were written by Honey's author and are not an independent
external suite; 23 tasks are enough to observe an effect, not a leaderboard.
- Judge scores are noisy. Objective tests are the stronger correctness signal.
- ESON's format benchmark measures serialization, not model quality. Its
separate upstream comprehension suite is not reproduced here.
- Neither benchmark recreates Peter's full graph, retries, audits, or context
history. Those can dominate the final bill.
- No saving is claimed for auth, money, migrations, deletes, or other
irreversible payloads; Peter keeps those explicit rather than compact.