Run claude-opus-5 (effort low) on the nine remaining splits: the only model that ever beat the top challenger has 1/10 coverage

[jonfroehlich]

claude-opus-5 at effort=low is the only model that has ever displaced gemini-3.1-pro
as top challenger — and it has run on 1 of 10 splits. That is enough coverage to make
the headline uncomfortable and not enough to change it.

Why this one matters more than the other coverage gaps

On annapolis, the only split where both ran:

model	P	R	F1
rampnet	0.973	0.738	0.839
claude-opus-5 (effort low)	0.572	0.605	0.588
gemini-3.1-pro-preview	0.613	0.527	0.567
claude-opus-5 (effort high)	0.430	0.656	0.520
claude-sonnet-5 (effort low)	0.589	0.381	0.463
claude-sonnet-5 (effort high)	0.506	0.415	0.456

docs/scoreboard.md reports RampNet's lead as +0.221 F1 over the best challenger, and
that best challenger is gemini-3.1-pro at a pooled 0.606. The model that beat it on the
one split where both ran is not in the pool. If the +0.021 annapolis edge held, Claude would
take the row and the lead would read ≈0.20.

That extrapolation is exactly what this issue exists to test, not to assume. annapolis is
one split, and it is the survey-grade Trimble MX7 city — the least representative rig in the
benchmark. Opus could just as easily regress on Mapillary 360s. Either answer is worth having:
a confirmed challenger changes the headline row, and a regression is a real result about what
annapolis's imagery flatters.

Scope

Run claude:claude-opus-5 --claude-effort low on the nine splits it has not seen:
richmond, bend, clovis, morgantown, paterson, gainesville, budapest_district5, sao_paulo,
manual_gold. 1,984 panoramas.

Deliberately not in scope:

• effort=high on anything. It is a measured negative: Opus 0.588 → 0.520 and Sonnet
  0.463 → 0.456, while spending ~127k extra thinking tokens on 125 panoramas. Effort moves
  the operating point (recall up, precision down); it does not raise the ceiling. Running it
  comprehensively would roughly double the bill to re-measure a result we have.
• claude-sonnet-5 low. At 0.463 it is mid-field, so it adds a row rather than a result.
  Worth completing later for the registry's sake; it is not what would change a headline.

Cost

From the measurement already in docs/model_comparison.md — claude-sonnet-5 at
effort=low bills 2,229 input / 39 output tokens per call, 0 thinking, six calls per
panorama — and the verified rates in scripts/model_comparison/pricing.py:

leg	rate (per MTok)	nine splits	batched (50%)
claude-sonnet-5 low	$2 / $10	~$57	~$29
claude-opus-5 low	$5 / $25	~$143	~$71

Opus is 2.5× Sonnet per token; token counts are assumed comparable, which is the one
soft number here. Sonnet's promotional rate ends 2026-08-31 ($3/$15 after, +50%); Opus
is not promotional so its estimate is stable.

Batching halves it and there is no reason not to — this is an offline benchmark leg with
no latency requirement.

Running it

Credentials are the same Vertex ADC the Gemini legs use — no Anthropic key, no new secret.
Two constraints are already documented in docs/model_comparison.md § "Claude on Vertex" and
neither needs re-discovering: structured_outputs is blocked by org policy so the tool must
stay un-strict, and forcing tool_choice suppresses thinking (harmless at low, where
there is none to lose).

# one split at a time; detections cache, so an interrupted run resumes for free
for split in richmond bend clovis morgantown paterson gainesville \
             budapest_district5 sao_paulo manual_gold; do
  python scripts/model_comparison/compare.py benchmark/$split \
      --models claude:claude-opus-5 --claude-effort low
done

# publish + verify (--publish-as, because effort is in the cache signature and both
# efforts of one model id would otherwise write the same filename)
python scripts/analysis/export_model_cache.py \
    --models claude:claude-opus-5 --claude-effort low \
    --publish-as claude-opus-5-effort-low
python scripts/analysis/export_model_cache.py --verify \
    --models claude:claude-opus-5 --claude-effort low \
    --publish-as claude-opus-5-effort-low

Gotchas that have already bitten this leg once

• The usage log is write-once. A cached re-run makes zero API calls and has no usage to
  report, so spend that is not captured on the first pass is unrecoverable. The four original
  legs' $28.82 exists only in console output for exactly this reason. Capture
  --usage-log output on the first run of each split.
• A parse failure costs a whole panorama, not a box. An exception in _raw_detect
  propagates past all six views. Watch the per-pano failure count rather than assuming a
  clean exit means clean coverage.
• Model enablement propagates unevenly in Vertex Model Garden — a valid request can 404
  for some hours after enabling. Smoke-test one panorama before launching nine splits.
• JPEG round-trip and decoding temperature are unpinned on the existing legs (see the
  caveats in docs/model_comparison.md). Match them — take the defaults — or the new splits
  are not comparable to the committed annapolis numbers.

Done when

• Nine splits run; detections published as claude-opus-5-effort-low__<split>.json and
  --verify reports IDENTICAL for 10/10 pairs
• Token counts and estimated cost in analysis_out/usage_log.jsonl, captured at run time
• rampnet/roster.py's entry updated (density measured across ten splits, not one)
• docs/scoreboard.md regenerated — Opus moves from the partial table into the headline,
  the PR-curve figure's point set, and the generalization figure. scoreboard.py --check
  green
• The best-challenger row and the +0.221 lead re-stated with whatever the answer is, in
  docs/scoreboard.md and docs/model_comparison.md
• Whether the annapolis edge replicated — stated either way

Related: #122 (the registry these legs are entered in, and where standing gets flipped
once this lands), #20 (the standing harness issue for newer models).

---

🤖 Generated with Claude Code (claude-opus-5[1m])

[jonfroehlich]

Smoke test passed; cost and wall-clock now measured rather than extrapolated

Ran 4 richmond panoramas through claude:claude-opus-5 --claude-effort low on
2026-08-18. Total spend so far: $0.31, logged to analysis_out/usage_log.jsonl at run
time.

Credentials work, with one wrinkle worth writing down: the leg needs
GOOGLE_CLOUD_PROJECT set in the environment. ADC alone is not enough — the first attempt
failed at prepare() with the detector's own error message, before making any API call, so
it cost nothing. gcloud config get-value project and the ADC file's quota_project_id
agree on this machine, so the fix is exporting one from the other rather than adding any new
secret. Model Garden enablement for Opus 5 is live — no 404.

Measured, from 18 real calls over 3 uncached panoramas:

	measured	previous estimate
calls per panorama	6	6
input tokens per panorama	12,186	13,374
output tokens per panorama	657	234
cost per panorama	$0.0767	$0.072
nine splits (1,984 panos)	~$152	~$143
wall clock per panorama	20.5 s	not estimated
nine splits, serial	~11.3 hours	not estimated

The cost estimate holds up (the body's ~$143 was derived from claude-sonnet-5's token
counts; Opus emits ~2.8× the output tokens, which the input-dominated bill mostly absorbs).
The wall clock is the number that was missing: score_model is a plain sequential loop,
so this is ~11,900 serial API calls and most of a day on one workstation.

That reframes the batching question in the issue body. Three options, with the transport
cost now visible:

approach	cost	wall clock	work needed
sync, serial (today's harness)	~$152	~11.3 h	none
sync + concurrency over the panorama loop	~$152	~1–2 h	small
Batch API	~$76	likely 1–6 h	new submit/poll/collect path

The harness has no batch path today — ClaudeDetector makes synchronous per-view calls
inside Detector.detect(sample), and batching inverts that control flow, so it is a code
change rather than a flag. It would also have to write into the same .model_cache keys so
export_model_cache --verify and every downstream scorer stay unchanged, and get its usage
accounting right on the first pass, because the log is write-once.

Worth noting for whoever picks this up: batching is a scheduling and billing tier, not a
different model — same weights, same parameters, same output distribution. There is no
accuracy cost to it. (These legs also do not pin temperature, so two sync runs are not
bit-identical to each other either; that is pre-existing, not a batch effect.)

The 4 smoke panoramas are cached, so none of them will be re-paid for in the real run.

---

🤖 Generated with Claude Code (claude-opus-5[1m])

[jonfroehlich]

Correction: the Batch API is not available on this transport at all

My previous comment listed "Batch API — ~$76, new submit/poll/collect path" as one of three
options. That option does not exist, and the correction matters because the table above
made it look like an engineering trade-off when it is a hard availability boundary.

The Claude leg runs through Vertex AI (docs/model_comparison.md § "Claude on Vertex":
same ADC, same project, same global location, no Anthropic API key). Anthropic's Message
Batches API is not served on Vertex — it is available on the first-party Claude API and
on Claude Platform on AWS only. The SDK's own client-side guard says so:

AnthropicError: The Batch API is not supported in the Vertex client yet

Probed 2026-08-18 against this project with a one-request batch. It was rejected before any
API call, so it cost $0 — and note the SDK does expose client.messages.batches on the
Vertex client, so an attribute check is a false positive. The guard fires at call time.

This is the same shape as the structured_outputs finding already in the doc: partner
models on Vertex get a subset of the first-party surface, and the subset is not discoverable
from the SDK's attribute surface.

Revised options

approach	cost	wall clock	work	blocked?
sync, serial (today's harness)	~$152	~11.3 h	none	no
sync + concurrency over the panorama loop	~$152	~1–2 h	small	no
Batch API on Vertex	—	—	—	impossible
Batch API by moving the leg to the first-party API	~$76	1–6 h	new transport + new credential	see below

The last row is the only route to the 50% discount, and it is not just an SDK swap:

• It needs an ANTHROPIC_API_KEY — a new secret, a separate billing account, and pricing
  that is not the Vertex partner pricing scripts/model_comparison/pricing.py records.
• It would silently mix provenance. Transport is not part of the detector's cache
  signature (provider/model_id/tile/max_edge/source_max_edge/views/prompt/
  effort/tool_choice), so annapolis's committed Vertex detections would still be found
  and the nine new splits would come from a different endpoint with nothing in the signature
  recording the difference. Any such run needs the transport pinned into the signature first,
  which orphans the paid annapolis cache.

Recommendation: concurrency on the existing Vertex path. It buys the wall-clock win —
the part that actually hurts — with no new credential, no billing change, and no provenance
mix. The $76 is not purchasable on this transport at any amount of engineering.

What the equivalence check becomes

The verification instinct still applies, but the target changes from "batch vs sync" to
"concurrent vs serial", and one layer of the naive version would fail for reasons unrelated
to correctness:

1. Cache-key identity — the load-bearing check, and fully deterministic. Assert the
   concurrent path writes exactly the keys the serial path writes. This depends on no model
   behaviour at all, and it is the failure with teeth: a key mismatch silently orphans paid
   detections and breaks export_model_cache --verify downstream.
2. Aggregate scores within sampling noise. Score both passes and compare P/R/F1.
3. Per-panorama byte-equality — a diagnostic, not a gate. These legs send no
   temperature, so they take the provider default and are sampled. Two serial runs
   are not byte-identical to each other either, so a strict equality gate would fail on
   pre-existing non-determinism rather than on anything concurrency did. Pinning
   --claude-temperature 0.0 for the A/B is possible but enters the cache signature, which
   would make the nine new splits a different configuration from the annapolis leg they are
   pooled with — so the pin cannot ship, only inform.

---

🤖 Generated with Claude Code (claude-opus-5[1m])

[jonfroehlich]

Detections for the eight remaining city splits are committed and pushed:
data/claude-opus-5-nine-splits-139 (e2f66fc). 984 panoramas — bend, budapest_district5,
clovis, gainesville, morgantown, paterson, richmond, sao_paulo — taking
claude-opus-5-effort-low to nine of ten splits. manual_gold was not run; see #144 for
why running it alone would not answer the question it appears to.

The annapolis edge did not replicate. It reversed.

Scored over the seven pooled city splits:

model	P	R	F1
RampNet	0.958	0.728	0.827
Gemini 3.1 Pro	0.656	0.573	0.608
YOLO11l (pano)	0.939	0.449	0.604
Claude Opus 5 (low)	0.573	0.614	0.588
Gemini 3.7 Flash	0.698	0.496	0.576

On annapolis alone Opus led gemini-3.1-pro by +0.021 F1. Pooled over seven splits it
trails by −0.021. The one split that made this issue worth opening was the one split that
flattered it — which is what the issue text predicted might happen, and it is the answer either
way.

The headline row does not change. gemini-3.1-pro remains the best challenger and RampNet's
lead stands at +0.219 F1 (0.827 vs 0.608). Opus slots fourth overall, second among chat
VLMs, below YOLO11l.

The real result is the precision/recall split

The F1 near-tie hides two models that are not alike:

• Precision −0.083 (0.573 vs 0.656) — this is the whole of the loss
• Recall +0.041 (0.614 vs 0.573) — and that makes Opus the highest-recall chat VLM in the
  benchmark, ahead of every Gemini leg

So "Claude loses" is true only on the aggregate. Against the repo's recall-first framing — a
false negative is permanent, a false positive is cheap — Opus is the strongest chat VLM
measured, and it loses on the axis we have said we care about least. That is worth stating
plainly wherever the ranking is quoted, because the ordering alone misrepresents it.

Per split, for the record:

split	P	R	F1
morgantown	0.628	0.670	0.649
paterson	0.752	0.559	0.642
bend	0.586	0.624	0.604
richmond	0.523	0.706	0.601
annapolis	0.572	0.605	0.588
clovis	0.528	0.574	0.550
gainesville	0.419	0.559	0.479
sao_paulo (not pooled)	0.442	0.498	0.468
budapest_district5 (not pooled)	0.425	0.340	0.378

budapest replicates as the collapse split for Opus too (0.378, its worst by a wide margin),
consistent with every other chat VLM and with the low reviewer confidence already recorded there.
paterson is the only split where Opus is precision-strong (0.752).

Spend: $70.41, and layer 1 did not record it

The run wrote no row to analysis_out/usage_log.jsonl. Recovered from Cloud Monitoring the
next day — 11,988,993 input / 418,503 output = $70.41 — and written up in
docs/model_comparison.md § "Cost accounting" (db7d0de). Reported in detail on #143, which had
assumed the paid-API half of the scheme needed no further work; it has a fifth gap.

Cheaper than the issue's ~$143 unbatched estimate because output ran lighter than assumed
(425 tokens/pano, no thinking at effort=low).

Still open on this issue

• rampnet/roster.py — density is still noted from annapolis alone (2.56 boxes/pano;
  measured over nine splits it is 2.61), and standing is still False
• docs/model_scoreboard.md regenerated — Opus is still in the partial table at annapolis
  only; scoreboard.py --check will fail until it moves
• The best-challenger sentence restated in docs/model_scoreboard.md and
  docs/model_comparison.md — the number barely moves (+0.221 → +0.219) but the claim
  "no model has displaced gemini-3.1-pro" is now tested rather than untested
• export_model_cache.py --verify run on the nine published files

Flipping standing=True changes the scored tables everywhere and touches the frozen
WITNESS_POOL_46, so that is a deliberate call rather than a regeneration — holding for review.

---

🤖 Generated with Claude Code (claude-opus-5[1m])