blindtest

You are paying for one model over another. Sit down for twenty minutes and find out whether you can even tell them apart.

blindtest runs a blinded discrimination trial between two language models on your own prompts, in your terminal, against any two OpenAI-compatible endpoints. It is the protocol breweries have used since the 1940s to test whether a cheaper ingredient changes the beer, applied to the question of whether a cheaper model changes your product. At the end you get an exact p-value, a confidence interval, and one of two answers:

• You could not tell them apart (say, 9 of 20, p = 0.19). On these prompts, with your own eyes, the difference you are paying for is not visible to you. The interval tells you how large a difference is still consistent with your data.
• You told them apart (say, 14 of 20, p = 0.0009). The difference is real for your workload, and now you know that instead of assuming it.

Both outcomes are worth having. What is not worth having is the current default: a strong opinion formed from two side-by-side samples, labels showing.

No accounts, no telemetry, no server. Your prompts and the responses never leave your machine except to go to the two endpoints you name.

Quickstart

export OPENAI_API_KEY=sk-...        # used for both sides by default
npx blindtest run gpt-5.4 gpt-5.4-mini --prompts prompts.txt

prompts.txt is your prompts, one per line (or a .jsonl of {"prompt": "..."}). Use prompts from your actual workload; that is the entire point. Twenty trials need twenty distinct prompts.

Different providers on each side, including local:

export BLINDTEST_A_KEY=sk-...       # key for side A
npx blindtest run gpt-5.4 qwen3:32b \
  --b-url http://localhost:11434/v1 \
  --prompts prompts.txt

No prompt file yet? --demo runs on a bundled set of generic engineering prompts, and the result screen will remind you that a demo run says nothing about your workload.

Before spending tokens, see what a design can detect:

npx blindtest plan

The protocol

Each trial shows you three responses to one of your prompts: two generated independently by one model, one by the other. You pick the odd one out. That is the whole task, repeated over --trials prompts.

This is the triangle test from sensory science, developed at the Carlsberg brewery in the 1940s (Helm and Trolle, 1946) and standardized since (ISO 4120). We use it instead of a plain A/B pairing for reasons that matter more for language models than for beer:

• Chance is 1/3, enforced by construction. The position of the odd response is drawn from a seeded RNG, fixed before any model is called, and never shown until the end. Under the null hypothesis that the responses carry no usable signal about their source, your pick is independent of that position, so P(correct) = 1/3 per trial exactly. The p-value's validity rests on the tool's randomization, not on assumptions about models or about you.
• Every trial carries its own yardstick. A model at temperature above zero is a distribution, not a fixed answer. The triangle shows you two independent draws from the same model next to one draw from the other, so you see within-model variation and between-model variation side by side. The task is literally "is the gap between models bigger than the gap within one", which is the actual question of interchangeability.
• No attribute to name. A classic forced-choice pairing needs a question like "which is more accurate", and the moment you name an attribute, halo effects answer instead (mostly: longer reads as better). The triangle asks only "which is different", so it measures discrimination without importing a theory of quality.

Balancing details, because they are where blind tests quietly break: which model appears twice alternates evenly across trials; the odd position is uniform; each prompt is used in exactly one trial (the tool refuses to reuse prompts, keeping trials independent); all responses are generated before the first trial is shown, so speed cannot leak; there is no feedback until the end, so you cannot calibrate on answers mid-run.

The statistics

Written to be checked. If you find an error, that is a bug report we want.

Null hypothesis. The judge cannot use the responses to locate the odd one; correct calls are Bernoulli(1/3), independent across trials.

Primary test. One-sided exact binomial: p = P(X >= k) for X ~ Binomial(n, 1/3), computed from the regularized incomplete beta, not a normal approximation. One-sided because below-chance triangle performance has no discrimination reading, which is also the standard treatment in sensory analysis.

Effect size, reported with every p-value.

• The proportion correct with a two-sided 95% Clopper-Pearson interval (Clopper and Pearson, 1934).
• The chance-corrected share of prompts where the difference was visible: pd = (3 Pc - 1) / 2, the guessing correction in the lineage of Abbott (1925), with the interval mapped through the same transform. 14/20 correct means pd = 55%: after removing lucky guesses, you saw a real difference on about half the prompts, with a 95% interval of 19% to 82%.
• Secondary, for people who speak signal detection: Thurstonian d' for the triangle decision rule (Frijters, 1979; Brockhoff and Christensen, 2010), with the interval mapped the same way. Labeled secondary because it assumes a perceptual model; pd assumes nothing.

A null result is a bound, not an equivalence proof. 9 of 20 gives p = 0.19 and a pd interval of 0% to 53%. The honest summary is "no evidence of a visible difference, and if one exists it shows on at most 53% of my prompts". The report prints exactly that upper bound and refuses to say "the models are the same".

Sample size. blindtest plan prints the exact power analysis. The interesting rows, at alpha 0.05:

trials	reject at	achieved alpha	power if Pc = 0.6	power if Pc = 0.7
20	11 correct	0.038	0.76	0.95
23	12 correct	0.048	0.84	0.98
30	15 correct	0.043	0.90	0.99

The default is 20 trials. If you want 90% power against a judge who genuinely sees a difference on 40% of prompts (Pc = 0.6), run 30. The "achieved alpha" column is there because the binomial is discrete: the real false positive rate of the printed decision rule is below the nominal 0.05, and pretending otherwise would be a small lie.

Peeking. With the default fixed design, the score is hidden until the run ends, and stopping early voids the analysis; the tool tells you so. If you want the right to stop early, pre-register it: --sequential switches the analysis to a mixture likelihood-ratio e-process for the 1/3 null (uniform mixture over p in (1/3, 1], in closed form via the incomplete beta). By Ville's inequality, the probability that the running e-value ever crosses 1/alpha under the null is at most alpha, so "stop the moment it crosses" is valid at any data-dependent stopping time. This construction is not ours: it descends from Wald (1945) through Ville (1939) and Robbins (1970) to the modern e-value literature (Ramdas, Grunwald, Vovk and Shafer, 2023; Grunwald, de Heide and Koolen, 2024). The tradeoff is bluntly stated: sequential runs stop fast when the difference is large (median 9 trials at Pc = 0.7 in simulation) and pay for that with less power than the fixed design at the same trial count.

The test suite does not take the theory's word for it. A Monte Carlo check simulates 20,000 null runs with an adversarial peeker who stops at the first crossing over a 200-trial horizon; the measured false positive rate at the shipped seed is 0.034, against the 0.05 bound.

Reproducibility. The design (trial count, arrangement seed, prompt hashes, normalization choices, endpoints) is written to disk and SHA-256 hashed before the first model call. The report carries the hash, every arrangement, every pick, and blindtest verify report.json recomputes the correctness flags, the counts, the p-value, and the hash from raw materials. This is tamper-evidence for auditing your own runs, not cryptography; a determined self-deceiver can always read the seed and cheat. The instrument is for people who want the answer.

Tells

A tell is anything that identifies the odd response without engaging with substance. We learned this the expensive way, watching users of an earlier in-house blind test identify a frontier model by sheer answer length, then, after length was equalized, by its em dashes. Tells are the failure mode of blind testing, and handling them honestly is most of this tool's design.

The uncomfortable truth first: a tell is real discrimination. If one model always writes longer, you genuinely can tell the models apart, and the test should and will say "distinguished". The problem is that it is usually not the discrimination you wanted to pay for. So blindtest does two things, and never silently:

1. It audits, always. After every run, mechanical strategies replay the trials with no language understanding at all:

• a length bot that picks the response whose length is least like the other two,
• a duplicate spotter that fires when two responses are near-identical (which exposes the third as the odd one),
• a formatting census per model: bullets, headings, bold, em dashes.

These appear next to your score. If you went 16/20 and the length bot went 15/20, the report says your discrimination is largely explained by length, in those words. You then know what you measured.

2. It normalizes, only if asked, and it discloses. Each option removes a channel, and each changes the question the run answers. The report states which channels were open.

option	what it does	what the question becomes
(default)	nothing removed	can you tell them apart at all
--strip-formatting	markdown structure and typography flattened; code blocks untouched	can you tell them apart by content
--length-band 80..120	both models instructed to answer inside the same word band	can you tell them apart at matched length, on a workload that includes that instruction
--truncate-match	responses cut to the shortest in the trial	crude; kills the channel and some content with it
--launder claude-haiku-4.5	a third model rewrites every response in neutral prose	can you tell the laundered forms apart. A null result no longer speaks to the originals: the laundering model may erase real differences or impose its own uniform errors. The report shouts this

Two channels are closed unconditionally, because no honest question needs them open: latency (all responses are prefetched before judging starts; timing appears only in the post-reveal appendix) and position (seeded uniform assignment, balanced duplication).

One channel cannot be closed by design: at temperature 0, the two same-model responses come out identical and the odd one is findable without reading. The tool refuses to run the triangle there; see the FAQ for the deterministic-endpoint story.

What a result means

• It is a statement about your workload, your eyes, and this week's model builds. It does not transfer to other prompt distributions, other judges, or the model behind the same name after the provider ships an update. Rerun; it is twenty minutes.
• Discrimination is not preference. This tool answers "can you tell them apart", not "which is better". Discrimination comes first because it gates preference: a preference between two models you cannot distinguish blind is, definitionally, noise about the models (it is usually a preference about labels and prices). If you cannot tell them apart, "which is better" has already been answered: neither, for you. Preference mode, with its own bias controls, is on the roadmap as the second instrument, not folded into the first.
• One pair per run. If you screen many model pairs and keep the best p-value, you are doing multiple comparisons; correct for it (the JSON report carries what you need) or better, treat screening runs as screening and confirm the winner with a fresh pre-registered run.

FAQ

Why three responses instead of two? With two responses and no labels there is nothing to ask: "which is from a different model" needs a reference for "different". The triangle supplies the reference inside the trial (the duplicated model) without ever labeling anything, keeps chance at 1/3 instead of 1/2, and shows you within-model variation so you do not mistake sampling noise for a model difference. The audio world's ABX solves the same problem with labeled references; the triangle does it without labels, which is one less thing to leak.

My workload runs at temperature 0. Then two draws from the same model are identical, a triangle trial degenerates into spot-the-duplicate, and the tool refuses to pretend otherwise. The design for deterministic endpoints is pair mode: one response from each model per trial, sides randomized, and you assign persistent nicknames, trying to be consistent; consistency with the true mapping is Binomial(n, 1/2) under the null, tested two-sided since systematically inverted labels are discrimination too. Its statistics are already implemented and tested in src/stats/; the interactive flow lands in v1.1.

Can an LLM be the judge instead of me? Not in v1, deliberately. It changes the claim from "I cannot tell them apart" to "model J cannot tell them apart", and LLM judges bring documented position and verbosity biases of their own (Zheng et al., 2023; Dubois et al., 2024). A machine-judge mode with those caveats printed on the result is planned; it is not the flagship.

My scores feel like guessing and it is uncomfortable. That discomfort is the data. Forced choice is the design: unsure guesses are exactly what a true null should produce, and the analysis prices them in. Skipping trials would not.

Does anything leave my machine? Prompts and responses go to the two endpoints you configured, and nowhere else. The report is a local file. There is no analytics, no version check, no network call you did not ask for. Read src/openai.ts; it is the only file that touches the network.

Why do you keep saying "on these prompts"? Because the strongest true claim is conditional. A tool that said more would be lying, and you would eventually notice.

Built by

blindtest is built by Finest, where the same discipline runs at production scale: proving, on your own workload, which cheaper model configurations hold your quality bar, before any traffic moves. The tool is MIT, standalone, and useful to people who never touch the product; that is by design.

References

• Helm, E. and Trolle, B. (1946). Selection of a taste panel. Wallerstein Laboratories Communications, 9, 181-194.
• ISO 4120. Sensory analysis. Methodology. Triangle test. International Organization for Standardization.
• Ennis, D. M. (1993). The power of sensory discrimination methods. Journal of Sensory Studies, 8, 353-370.
• Frijters, J. E. R. (1979). The paradox of discriminatory nondiscriminators resolved. Chemical Senses and Flavour, 4, 355-358.
• Thurstone, L. L. (1927). A law of comparative judgment. Psychological Review, 34, 273-286.
• Green, D. M. and Swets, J. A. (1966). Signal Detection Theory and Psychophysics. Wiley.
• Clopper, C. J. and Pearson, E. S. (1934). The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26, 404-413.
• Abbott, W. S. (1925). A method of computing the effectiveness of an insecticide. Journal of Economic Entomology, 18, 265-267.
• Bi, J. (2015). Sensory Discrimination Tests and Measurements, 2nd ed. Wiley.
• Brockhoff, P. B. and Christensen, R. H. B. (2010). Thurstonian models for sensory discrimination tests as generalized linear models. Food Quality and Preference, 21, 330-338.
• Wald, A. (1945). Sequential tests of statistical hypotheses. Annals of Mathematical Statistics, 16, 117-186.
• Ville, J. (1939). Etude critique de la notion de collectif. Gauthier-Villars.
• Robbins, H. (1970). Statistical methods related to the law of the iterated logarithm. Annals of Mathematical Statistics, 41, 1397-1409.
• Ramdas, A., Grunwald, P., Vovk, V. and Shafer, G. (2023). Game-theoretic statistics and safe anytime-valid inference. Statistical Science, 38, 576-601.
• Grunwald, P., de Heide, R. and Koolen, W. (2024). Safe testing. Journal of the Royal Statistical Society, Series B, 86.
• Zheng, L. et al. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. NeurIPS Datasets and Benchmarks.
• Dubois, Y., Galambosi, B., Liang, P. and Hashimoto, T. (2024). Length-controlled AlpacaEval: a simple way to debias automatic evaluators. arXiv:2404.04475.

MIT. Issues and replication reports welcome. Citing a result: name the design hash from the report, the trial count, and the score; citing the tool: blindtest, github.com/finest-ai/blindtest.