bench: land cost-quality baseline with reliable result capture

[jamestexas]

What changed

• restores the historical benchmarks/cost-quality/ harness, prompt corpus, archived first run, and candid results post-mortem
• parses Claude JSON as a complete object even when pretty-printed or surrounded by log output
• records malformed or structurally invalid successful stdout as an explicit error instead of a zero-cost success
• stops treating legitimate responses that mention error early as failed answers
• adds focused Python regression tests for both review findings
• refactors benchmark orchestration enough to clear duplicate_definitions and fan_out_skew without expanding the smell baseline

Validation

• python3 -m unittest discover -s benchmarks/cost-quality -v
• task smells with the fixed local LLO 0.18.1 binary
• full task ci in the normal pinned-Leyline environment, including race tests and install:verify

The archived results remain a historical baseline, not a claim that the first run is decision-grade; RESULTS.md documents those limitations.

[Copilot]

Pull request overview

This PR restores the benchmarks/cost-quality/ benchmark harness and its first historical run artifacts (JSONL + post-mortem) from an orphaned branch, so a future “proper rerun” has an in-repo baseline to build on.

Changes:

• Adds the cost+quality benchmark runner (bench.py) plus inputs (prompts.json, mcp.json) and usage docs (README.md).
• Checks in the first captured run output (results/run_20260513.jsonl) and a write-up explaining why the numbers shouldn’t be used yet (RESULTS.md).
• Updates .beads/beads.jsonl with the latest bead record for mache-7555da.

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
benchmarks/cost-quality/results/run_20260513.jsonl	Archived JSONL output from the first benchmark run (baseline artifact).
benchmarks/cost-quality/RESULTS.md	Post-mortem describing why the run isn’t a meaningful mache evaluation yet.
benchmarks/cost-quality/README.md	How to run the benchmark and interpret outputs; documents provenance/attribution.
benchmarks/cost-quality/prompts.json	Prompt battery used by the benchmark runner.
benchmarks/cost-quality/mcp.json	MCP server config template used by claude in “mache” mode.
benchmarks/cost-quality/bench.py	Benchmark runner + heuristic quality scorer (resume-safe JSONL streaming).
.beads/beads.jsonl	Updates bead record(s) (notably mache-7555da) to the latest state/comments.

Suppressed comments (1)

benchmarks/cost-quality/bench.py:108

• If parse_claude_json returns {}, run_claude currently proceeds as if it succeeded and writes a row with zero tokens/cost and empty response. This makes format regressions in the claude CLI look like valid (but empty) benchmark results rather than hard failures.

    data = parse_claude_json(result.stdout)
    usage = data.get("usage", {})

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[jamestexas]

Leaving this as a draft, but flagging a landmine before it ever gets merged: the branch carries .beads/beads.jsonl at a 2026-08-11 snapshot (565 beads). Main is now at 599, and 34 beads exist on main that are absent here — beads.jsonl is a whole-file export, so merging as-is silently reverts all 34 rather than merging them.

Same defect just closed #609 (that one had only the stale export, so there was nothing to save; this one has 826 lines of real benchmark harness worth keeping).

Suggested fix when this comes off draft: drop .beads/beads.jsonl from the branch entirely (git checkout main -- .beads/beads.jsonl) so the PR carries only benchmarks/cost-quality/. The bead state should ride a fresh export, not a three-week-old one.

Not touching the branch since it's yours and still draft.