claude-opus-5 on manual_gold: recommend NOT running it alone, because the top challenger is not on that split either

[jonfroehlich]

claude-opus-5 at effort=low now covers nine of the ten benchmark splits (#139). The
missing one is manual_gold, and this issue records why it was left out and why running it
alone would not answer the question it appears to answer.

Recommendation: do not run it on its own

manual_gold is the benchmark's most valuable split — 1,000 panoramas, the only
independently-labeled ground truth we have. It is also the most expensive, and the argument
against a Claude-only run is not the cost.

The top challenger is not on that split either. Of the roster, only nine legs have a
manual_gold row:

Molmo2-8B, gemini-3.7-flash, owlv2-large-patch14-ensemble, grounding-dino-base,
Qwen3-VL-32B, Qwen3-VL-8B, y11l_pano, y11x_pano_h200, y26_pano

gemini-3.1-pro-preview and gemini-3.6-flash are absent — already noted as a caveat in
docs/model_comparison.md. gemini-3.1-pro is the model the headline "+F1 over best
challenger" figure is measured against.

So a Claude manual_gold run would produce a number whose only chat-VLM peer on that split is
gemini-3.7-flash — not the leg the claim is about. It would cost real money to obtain a row
that cannot be compared to the thing it needs comparing to. It is a pair of runs or neither.

Cost, if it is done

Input tokenization is deterministic, so the input half is exact: an Opus pano is 12,186
tokens (6 views × 2,031). Output is measured at 425 tokens/pano from the nine-split leg
(11,988,993 in / 418,503 out over 984 panos, Cloud Monitoring 2026-08-19).

leg	input	output	cost
claude-opus-5 low, 1,000 panos	12,186,000	~425,000	~$71
gemini-3.1-pro-preview, 1,000 panos	—	—	~$18.50 (already costed in the doc)
both, so the comparison is real			~$90

Batching halves the Claude half — this is offline, so there is no reason not to. Call it
~$54 batched for the pair.

What would make this worth doing

• The headline claim comes to rest on manual_gold specifically, or
• A reviewer asks for the top-two challengers on the only independently-labeled split, or
• gemini-3.1-pro-preview gets its manual_gold row for another reason, at which point
  Claude's becomes cheap to add and immediately meaningful.

Until one of those, nine splits is the honest coverage and the gap is stated next to the number
rather than quietly left blank.

Done when

• Either both legs run and docs/model_scoreboard.md gains a real manual_gold comparison,
• or this issue is closed as deliberately-deferred and the omission stays documented in
  docs/model_scoreboard.md and docs/model_comparison.md

Related: #139 (the nine-split leg), #122 (the registry), #20 (standing harness issue for newer
models).

---

🤖 Generated with Claude Code (claude-opus-5[1m])