How much of a SWE-bench leaderboard gap is real?
On SWE-bench Verified, 132 of the 133 gaps between adjacent leaderboard entries are smaller than the smallest difference 500 tasks can detect. Under an exact paired test with a family-wise correction, none of them is significant. The median gap between neighbours is 0.40 accuracy points; 28 pairs are exact ties.
This repository is the arithmetic behind that sentence. It reads the published per-instance
outcomes from SWE-bench/experiments — 324 submissions
across six splits, every one of them a list of which tasks that system actually resolved — and asks
three questions the leaderboard does not answer.
Pure standard library. No scipy, no numpy, no API key, no GPU. The point is that a leaderboard's resolving power is arithmetic anyone can re-run, not a library call nobody checks.
Run on 2026-08-21 against SWE-bench/experiments at main. Every number below is reproduced by
the command in Reproducing.
| Split | Systems | Tasks | Median 95% CI | Adjacent pairs significant (uncorrected / Holm) | Top-N indistinguishable from the leader | MDD |
|---|---|---|---|---|---|---|
| verified | 134 | 500 | 8.35 pp | 4 / 133 · 0 / 133 | 8 | 5.37 pp |
| lite | 84 | 299 | 10.69 pp | 1 / 83 · 1 / 83 | 4 | 7.07 pp |
| test | 24 | 1839 | 3.87 pp | 14 / 23 · 11 / 23 | 1 | 2.61 pp |
| multimodal | 12 | 301 | 11.19 pp | 0 / 11 · 0 / 11 | 5 | 6.45 pp |
| multilingual | 13 | 301 | 10.56 pp | 1 / 12 · 1 / 12 | 6 | 7.27 pp |
| bash-only | 40 | 500 | 8.24 pp | 6 / 39 · 5 / 39 | 6 | 5.04 pp |
MDD = minimum detectable difference: the smallest accuracy gap a paired design at that n can find at 80% power and a two-sided α of 0.05, given the observed discordance between neighbours.
The test row is the control, and it is the reason to trust the rest. With n = 1839 instead of
500, roughly half the adjacent pairs do separate — 11 of 23 survive Holm. The method is not a
machine that prints "not significant" regardless of input; it prints that on Verified because
Verified is 500 tasks and the field is packed into the top ten points.
The leader (sonar-foundation-agent_claude-opus-4-5, 396/500 = 79.2%) is not provably ahead of
ranks 2 through 8. Rank 2 is an exact tie at 396/500. The first entry it beats at even an
uncorrected 0.05 is rank 9, 3.6 points below it:
rank 2 Δ=+0.0pp p=1.0000 livesweagent_claude-opus-4-5
rank 3 Δ=+0.4pp p=0.9007 trae_doubao_seed_code
rank 4 Δ=+1.6pp p=0.2430 openhands_claude-opus-4-5
rank 5 Δ=+1.8pp p=0.2529 livesweagent_gemini-3-pro-preview
rank 6 Δ=+2.4pp p=0.1550 epam-ai-run-claude-4-sonnet
rank 7 Δ=+2.4pp p=0.1337 atlassian-rovo-dev
rank 8 Δ=+2.8pp p=0.1143 ACoder
rank 9 Δ=+3.6pp p=0.0247 warp ← first p ≤ 0.05
The four adjacent pairs that clear an uncorrected 0.05 are all at ranks 125–132 — the 2023–2024 RAG baselines, where the field is genuinely spread out. In the entire top 100, not one neighbouring pair separates.
A task that every system solves, or that no system solves, cannot separate anyone. On Verified, across all 134 submissions: 32 tasks were solved by nobody, 8 by ≥95% of submissions. The effective discriminating set is 427 of 500 (85.4%). On Lite it is 209 of 299 (69.9%); on multimodal, 155 of 301 — half the multimodal set is decided before any system runs.
The analysis above ignores run-to-run variance entirely, because a SWE-bench submission is
one scored run and the data contains none. tau-bench
(MIT) ships historical_trajectories/ — every task's outcome on every trial for two of
its seven leaderboard entries. That is a repeated-measures design, and it supports the
question a single-run board cannot ask: how much of a score is the system, and how much is
the dice.
Parity first. All four published pass^k rows reproduce exactly from those files, to
three decimals, before any of the numbers below are computed.
| tasks | trials | flips | ICC(1) | pass^1 per trial | spread | |
|---|---|---|---|---|---|---|
| airline gpt-4o | 50 | 4 | 52% | 0.405 | .42 .44 .40 .42 | 0.040 |
| airline sonnet-3.5-new | 50 | 8 | 64% | 0.465 | .50 .40 .44 .48 .46 .46 .42 .52 | 0.120 |
| retail gpt-4o | 115 | 4 | 43% | 0.530 | .617 .617 .600 .583 | 0.035 |
| retail sonnet-3.5-new | 115 | 8 | 58% | 0.456 | .730 .704 .704 .687 .704 .609 .748 .652 | 0.139 |
flips is the share of tasks that neither always pass nor always fail across that
system's own trials. Between 43% and 64% of tasks. Over half the airline set is neither
solved nor failed by a given agent — it is tossed for.
ICC(1) puts a number on it: 0.41 to 0.53. Less than half the total variance sits between tasks. The rest is the same agent, the same task, a different run.
The published gap is smaller than one system's own range, in both environments. Airline
#1 − #2 is 0.040 against sonnet's own per-trial spread of 0.120. Retail is 0.088 against
0.139. The gap between first and second place is a third of the distance the winner travels
against itself.
What one run resolves. Restricted to trial 0 — the shape a single-run leaderboard forces on you:
| pass^1 | 95% Wilson | exact McNemar | MDD at 80% power | |
|---|---|---|---|---|
| airline, gpt-4o vs sonnet | .420 / .500 | [.294,.558] / [.366,.634] | b=7 c=11, p=0.481 | 23.77 pp |
| retail, gpt-4o vs sonnet | .617 / .730 | [.526,.701] / [.643,.803] | b=9 c=22, p=0.029 | 13.56 pp |
The airline MDD of 23.77 pp is worth sitting with: the entire published airline leaderboard spans 22.5 to 46.0 pass^1, which is 23.5 points. On a single run this benchmark can barely separate its best entry from its worst.
Retail is the control, and it is why this is not a machine for printing "not significant": at n=115 the same pair does separate, p=0.029.
More trials do not fix it. The between-task component does not shrink with reruns. With the within-task term driven to zero, 50 airline tasks still leave a half-width of ±10.2 pp and 115 retail tasks ±6.1 pp. Only more tasks move those.
Three limitations of tau-bench, stated as such. Four of the seven Retail rows on the
published board are ?? — not reported at all. Per-task data ships for two of seven
entries. And the two that ship it were run with different trial counts — gpt-4o 4,
sonnet 8 — then placed in one table as though they were the same measurement.
python3 src/tau.py --root ./tau-data --env airline --fetch
python3 src/tau.py --root ./tau-data --env retailCI runs both environments from a fresh fetch and fails if parity with the published board breaks, if a flip rate collapses, or if a published gap stops being smaller than the winner's own spread.
- How wide is each entry's interval, at the benchmark's actual n? Wilson score interval, chosen over Wald because Wald misbehaves near 0 and 1 and a leaderboard's ends are where the interesting entries sit.
- Are adjacent entries distinguishable? Exact McNemar on the paired 2×2 table, then Holm–Bonferroni across the whole ladder. Paired, because both systems are scored on the same tasks and the concordant cells carry no information about the difference — comparing two independent CIs throws that pairing away and is strictly weaker. Exact rather than chi-square, because adjacent entries routinely produce fewer than 25 discordant tasks, where chi-square's nominal 0.05 is not 0.05. Holm, because a 134-entry ladder invites 133 comparisons and an uncorrected 0.05 manufactures about one "significant" gap per twenty out of nothing.
- How many tasks discriminate at all? Count the tasks solved by everyone or no one, and report what is left.
- This measures the leaderboard's resolving power, not any system's quality. "Not distinguishable at n=500" is not "the same". Two systems tied here may differ in ways 500 tasks cannot see, and a 1.6-point gap that fails a significance test may still be a real 1.6 points.
- It ignores run-to-run variance entirely. Each submission is a single run, scored once. The real interval around a leaderboard entry is wider than what is computed here, because sampling noise in the agent's own execution is not in the data at all. Every conclusion is therefore conservative in the direction of the leaderboard looking better than it is.
- Submissions are not independent draws. Many share a base model, a scaffold, or both. Holm controls family-wise error without assuming independence, which is why it is used, but the ladder's structure is still not a clean experimental design.
- Ten
multimodaland sevenbash-onlysubmissions are excluded by name, because they report a resolved count with no instance list. A paired test over unknown per-task outcomes would be arithmetic on invented data. The excluded names are printed on every run and stored in the output JSON. testandliteuniverses are short of their published sizes — 1839 of 2294 and 299 of 300. Those instances appear in no submission's report. Denominators use the observed universe and the run says so rather than assuming the published n.
No dependencies, no data checked in — the outcomes come from upstream, so the numbers in this README are only as stale as your clone of it. Python 3.11+.
git clone --depth 1 --filter=blob:none --sparse https://github.com/SWE-bench/experiments.git experiments
git -C experiments sparse-checkout set evaluation # ~120 MB
python3 src/test_stats.py # 33 checks on the statistics
python3 src/resolve.py --root ./experiments --split verifiedCI runs exactly that on every push and fails if the headline claim stops holding — a Holm-significant adjacent pair appearing, or more than one gap rising above the detection floor.
--split takes any of verified lite test multimodal multilingual bash-only. Output goes to
out/<split>.json with every per-entry interval, every adjacent pair's exact p-value and discordant
counts, and the leader-versus-rank-N ladder.
src/stats.py Wilson interval, exact McNemar, Holm step-down, paired MDD, Gwet's AC1,
and for repeated measures: pass^k, flip rate, ICC(1), trials-to-precision
src/load.py both result schemas, and the exclusion rule for count-only submissions
src/resolve.py the three questions, one split at a time
src/tau.py tau-bench: what a leaderboard hides when the same task is run 4-8 times
Agent benchmarks are reported as a single number and read as a ranking. The number arrives with no interval, the ranking arrives with no test, and the gap between rank 3 and rank 4 is discussed as though it means something. On the most-cited agent benchmark in the field, at its published size, it does not.
The fix is not a bigger leaderboard. It is reporting an interval next to the score, and stating the smallest difference the benchmark can resolve — so that a 0.4-point improvement is read as what it is.
MIT.