Summary
The docs already note that LLM judges can favor longer/verbose answers. This is a reproducible quantification of that (and of self-preference) computed on the shipped GPT-4 pairwise judgments, offered as a concrete caveat that could be linked from the llm_judge docs.
Numbers — from the shipped gpt-4_pair.jsonl, general templates, n=2,999
- Where the two answers differ in length, the longer answer wins 68.0% of the time (z=18.3, p≈0). Winners average ~1,080 chars vs losers ~950.
- Self-preference: GPT-family vs non-GPT, the GPT side wins 71.5% (z=22.8). GPT-4 judging its own answers: 82/94 = 87.2%, and here length is balanced (GPT answer longer only 51.8% of the time) — so this isn't just the length effect wearing a disguise.
- Position: the answer shown first wins 52.6% (p≈1e-4); 12.9% of paired verdicts flip when the two answers swap seats.
Kept honest
The length result is a correlation, not proven causation — longer answers can also be better, and this observational data can't fully separate the two. The self-preference result is sturdier because length was balanced in those matchups. So this isn't "the judge is broken"; it's "a raw win rate is entangled with length and self-preference by double-digit points, and a reader can't tell how much of a score is which."
Suggestion
A short documented caveat (or a linked reproduction) for anyone quoting raw MT-Bench win rates, plus the usual mitigations: length-match the answers, use a disinterested judge or an ensemble, and always run both orderings and report the flip rate.
Reproduction is deterministic on the shipped file. Happy to share the script or open a PR adding a bias-check under llm_judge.
Summary
The docs already note that LLM judges can favor longer/verbose answers. This is a reproducible quantification of that (and of self-preference) computed on the shipped GPT-4 pairwise judgments, offered as a concrete caveat that could be linked from the
llm_judgedocs.Numbers — from the shipped
gpt-4_pair.jsonl, general templates, n=2,999Kept honest
The length result is a correlation, not proven causation — longer answers can also be better, and this observational data can't fully separate the two. The self-preference result is sturdier because length was balanced in those matchups. So this isn't "the judge is broken"; it's "a raw win rate is entangled with length and self-preference by double-digit points, and a reader can't tell how much of a score is which."
Suggestion
A short documented caveat (or a linked reproduction) for anyone quoting raw MT-Bench win rates, plus the usual mitigations: length-match the answers, use a disinterested judge or an ensemble, and always run both orderings and report the flip rate.
Reproduction is deterministic on the shipped file. Happy to share the script or open a PR adding a bias-check under
llm_judge.