A benchmark for how language models handle belief, knowledge, reasoning, defeaters, and epistemic luck.
🌐 Live app: epistemically.com
Epistemically is a 200-case evaluation suite and interactive dashboard for studying how language models classify the epistemic status of a claim.
Most benchmarks ask whether an answer is correct. Epistemically asks a different question:
Does the model correctly distinguish belief, truth, justification, knowledge, rational inference, defeaters, and epistemic luck?
The benchmark evaluates model outputs under explicit operational definitions. It does not claim that language models literally believe, know, or possess epistemic agency.
| Cases | 200 |
| Models | 17 |
| Providers | 3 |
| Modules | 4 |
| Prompt conditions | 2 |
| Model-case-condition results | 6,800 |
| Output format | Structured JSON |
| Scoring | Exact-label components and deterministic coherence checks |
Each case is evaluated independently in a fresh model call. Models do not see earlier cases, expected answers, scores, or feedback.
A model can reach the right conclusion for the wrong epistemic reason.
A claim can be:
- true without being known
- justified but false
- justified and true but still not knowledge because of luck
- rationally defeated by new evidence
- accepted for practical reasons without being believed
- inferred validly from false premises
Epistemically makes these distinctions measurable. It produces an interpretable profile of where each model succeeds, where it fails, and what kinds of mistakes it makes.
| Module | Cases | What it tests |
|---|---|---|
| Belief, acceptance & knowledge | 60 | Belief, truth, pragmatic acceptance, reasons, justification, and knowledge |
| Defeaters | 50 | Rebutting, undercutting, higher-order, and irrelevant information, plus coherence among strength, support, and belief revision |
| Rational reasoning | 30 | Logical status, deductive patterns, invalid inference, and rational constraints |
| Epistemic luck | 60 | Knowledge controls, lucky guesses, intervening luck, environmental luck, and no-knowledge controls |
Defeater cases include two components computed from the model's own labels:
- Whether defeater strength and comparative support describe the same evidential state
- Whether the reported belief revision follows from the reported support relation
These checks are deterministic. They do not rely on another language model as a judge.
Epistemically evaluates the same 200 cases under two prompt conditions:
- Label-only: scenario, target, question, required fields, and allowed labels
- Full rubric: the same case plus definitions and decision guidance
This creates a paired experiment across 17 models and 200 matched cases per condition.
Across all 17 models:
| Result | Guidance gain |
|---|---|
| Overall average | +4.25 pp |
| Defeaters | +14.20 pp |
| Epistemic luck | +1.75 pp |
| Belief, acceptance & knowledge | +0.39 pp |
| Rational reasoning | +0.39 pp |
- Full-rubric guidance produced 542 rescues and 234 regressions.
- Model-level effects ranged from nearly unchanged to gains above 14 percentage points.
- Model rankings changed between conditions, showing that prompt design can affect comparative results.
- The largest gains were concentrated in Defeaters, where models had to apply a structured taxonomy of evidence and belief revision.
These are benchmark-specific findings, not general claims about model intelligence.
| Provider | Models |
|---|---|
| OpenAI | gpt-4o-mini, gpt-5-mini, gpt-5.6-luna, gpt-5.6-terra, gpt-5.6-sol |
| Anthropic | claude-haiku-4-5, claude-sonnet-5, claude-opus-4-8, claude-opus-5, claude-fable-5 |
gemini-2.5-flash-lite, gemini-2.5-pro, gemini-3.1-flash-lite, gemini-3.5-flash-lite, gemini-3.5-flash, gemini-3.6-flash, gemini-3.1-pro-preview |
The live app contains current rankings, provider comparisons, uncertainty intervals, and case-level results.
| View | What it provides |
|---|---|
| Overview | Leaderboards, provider comparisons, ranking uncertainty, and detailed module, family, and field breakdowns |
| Prompt Guidance | Guidance gain, confidence intervals, practical-effect bands, rank movement, win/tie/loss, rescues, regressions, and selected permutation tests |
| Model Profile | Full-rubric, Label-only, and comparison profiles with benchmark medians, confidence intervals, family and field performance, and hardest cases |
| Model Comparison | Same-case comparisons between models, prompt conditions, or differences in Guidance gain |
| Failure Gallery | Recurring errors, difficult families, field mistakes, label confusions, parse failures, rescues, and regressions |
| Case Explorer | Individual responses, expected labels, field-level correctness, raw outputs, and side-by-side model or prompt-condition comparisons |
The footer includes a concise methodology and run-health details.
Models return fixed labels for each required field. Labels are scored by exact match after light normalization.
case score = weighted correct components / total component weight
This keeps scoring transparent and auditable. Every point traces to a specific component on a specific case.
Brief explanations are collected for diagnosis but are not scored.
- End-to-end score: performance including whether the structured response was usable
- Valid-output accuracy: semantic performance when a usable structured response was returned
- Structured-output reliability: share of responses that met the required output structure
- Fully correct rate: share of cases with every scored component correct
- Equal-weight module score: average that gives each module equal influence
- Field and family accuracy: exact-label performance at more detailed levels
Epistemically uses case-level resampling so uncertainty reflects variation across benchmark cases.
- Percentile bootstrap confidence intervals summarize model and module performance.
- Pairwise model comparisons use paired bootstrap differences on the same cases.
- Prompt-condition effects use 2,000 paired-bootstrap resamples and 95% percentile intervals.
- A ±1 percentage-point practical band distinguishes clear effects from small or uncertain differences.
- Selected prompt-condition comparisons use exploratory, two-sided paired permutation tests with 10,000 sign flips.
- Permutation p-values are unadjusted.
- Win/tie/loss counts summarize per-case score differences.
- Rescues and regressions track changes in fully correct cases.
Effect sizes and confidence intervals are treated as the main evidence. P-values are secondary.
- Every case is evaluated independently.
- Runs use temperature 0.
- Expected answers and scoring are deterministic.
- The dashboard reads committed result files and does not call provider APIs.
- API keys are not required to view or run the app locally.
- Run-health checks surface duplicates, missing cases, parse failures, and result coverage.
Clone the repository:
git clone https://github.com/ACSanders/epistemically-eval.git
cd epistemically-evalCreate and activate a virtual environment:
python -m venv .venvWindows PowerShell:
.\.venv\Scripts\Activate.ps1macOS or Linux:
source .venv/bin/activateInstall dependencies:
pip install -r requirements.txtLaunch the app:
streamlit run app.pyThe dashboard runs from committed result files. No API keys are needed.
API keys are only required for new evaluations.
Create a local environment file:
cp .env.example .envWindows PowerShell:
Copy-Item .env.example .envAdd the provider keys you plan to use:
OPENAI_API_KEY=your_openai_key_here
ANTHROPIC_API_KEY=your_anthropic_key_here
GEMINI_API_KEY=your_gemini_key_here
Validate the case file:
python scripts/validate_cases.py data/cases/user_cases_draft.jsonlRun an evaluation:
python scripts/run_eval.py \
--cases data/cases/user_cases_draft.jsonl \
--model gpt-5-mini \
--output data/results/user_cases_results_gpt-5-mini.csvDo not commit .env.
epistemically-eval/
├── app.py
├── data/
│ ├── cases/
│ ├── results/
│ └── condition_results/
├── docs/
├── scripts/
├── src/
│ └── epistemically/
│ ├── analysis.py
│ ├── bootstrap.py
│ ├── case_intel.py
│ ├── dataset.py
│ ├── metrics.py
│ ├── model_profile.py
│ ├── overview.py
│ ├── prompt_guidance.py
│ ├── scoring.py
│ └── runners/
└── tests/
- Each model-condition result is based on one temperature-0 run, so the benchmark does not measure variation across repeated generations.
- Results depend on the benchmark's operational definitions and prompt wording.
- Full-rubric guidance changes several prompt elements at once, so a gain cannot be attributed to one specific definition or rule.
- Exact-label scoring is transparent but strict.
- Some cases, especially environmental-luck cases, remain philosophically contestable.
- Results describe performance on this benchmark and should not be treated as universal capability rankings.
MIT License.