Spoiler-free before. Every twist after.
Twistify is a movie app with a rule that's actually enforced, not just promised: the plot never leaves the server until you say you've already seen it. Underneath, an evaluation harness measures whether that promise holds — with numbers, not a self-awarded green badge.
git clone https://github.com/serpeigd/Twistify.git
cd Twistify
pip install fastapi "uvicorn[standard]" pydantic pyyaml
python webapp/app.py
# open http://127.0.0.1:8000Pick a researched movie (Sixth Sense, Fight Club, Get Out, Parasite, The Prestige, Se7en, or Arrival), read the spoiler-free entry, and when you're ready, open the curtain.
- The spoiler partition is a server-side property, not a UI promise.
Post-viewing content isn't sent to the browser until the client declares
seen=true— opening devtools reveals nothing. It's not CSS hiding a<div>. - Every factual claim says whether it has a source or not. No faking
source_ids when there's no real retrieval behind it. The gap is shown, not disguised. - The spoiler-leak detector itself is measured, not assumed. There's an
evals harness (
evals/) that calibrates the judge against planted leaks and reports its real recall — including the uncomfortable case where the cheap judge fails (see Results). - Filters that actually mean something. Themes (identity, obsession, class and power…) that group several movies for real, not a one-off tag per title.
Backend: Python 3.12 · FastAPI · Pydantic v2 (typed data contracts, not loose dicts) · pytest (evals harness, runs with no network, no API key)
Frontend: vanilla HTML/CSS/JS — no framework, on purpose: the app is small enough that a framework would be cost without benefit, not "doesn't know how to use one."
AI / evaluation: Anthropic Claude (tool use / structured output for the baseline generator, no markdown parsing) · a custom evals harness design (leakage / grounding / richness) with a calibrated judge, verified against planted leaks.
CI: GitHub Actions runs the 8 tests on every push (see badge above).
| Piece | What it is |
|---|---|
webapp/ |
FastAPI + vanilla JS. Serves the catalogue, runs the spoiler gate, comments (edit/delete with no accounts, anonymous per-browser token). |
content/researched/*.json |
7 hand-researched entries (with cited sources: Wikipedia, Hollywood Reporter, No Film School…), not generated by an unverified LLM. |
src/preshow/ |
Data contracts (Pydantic) for both the researched content and the measurement harness. |
evals/ |
The real experiment: leakage/grounding/richness metrics, calibrated judge, 20-title stratified dataset. |
| What | Status |
|---|---|
| Twistify app (catalogue, spoiler gate, filters, comments) | ✅ 7/20 entries researched |
| Browse catalogue (TMDB posters, live search, ES/EN) | ✅ 20/20 have posters, search reaches all of TMDB |
| Offline evals harness | ✅ 8 tests passing |
| Spoiler ground truth (20 titles) | ✅ 20/20, researched with cited sources |
| Baseline generator (no retrieval) | ✅ two providers — Anthropic (paid) and Groq (free tier, no card) |
| Judge calibration (offline + real spoiler reviews) | ✅ recall=0.0 confirmed twice — SubstringJudge needs replacing, not just calibrating |
| Measure the baseline over the 20 titles | ✅ done — see numbers and caveats below |
| Retrieval (TMDB/OMDb/Wikipedia) + verifier | ⬜ next milestone |
The part you don't see in the screenshots is what backs the app's promise: a system that measures, instead of promising, three things per entry:
leakage_rate— did any spoiler slip into the pre-viewing content?grounded_fact_rate— how many claims carry a real source?richness— how much does it actually say? (an empty output scores perfectly on the first two — that's why it's never reported without this one)
python -m pytest tests/ -q # 8/8, no network, no API key
python evals/run_eval.py --generator baseline-groq # free tier, no card
python evals/run_eval.py --generator baseline # or the paid Anthropic version| Metric | Mainstream | Long-tail | Overall |
|---|---|---|---|
leakage_rate |
0.0 | 0.0 | 0.0 |
grounded_fact_rate |
0.0 | 0.0 | 0.0 |
richness (claims/case) |
6.0 | 6.0 | 6.0 |
Read this table with its caveats, not instead of them:
leakage_rate = 0.0is not a safety result — it's the judge's blind spot.SubstringJudgewas calibrated offline at recall = 0.0: it only catches verbatim spoiler phrases, never a paraphrase. A 0.0 leakage rate here most likely means the judge failed to see leaks that are actually there, not that the baseline is safe. Trusting this number without the calibration note next to it is exactly the mistake this project exists to avoid.grounded_fact_rate = 0.0is a real, expected finding. The baseline is given no retrieval corpus (corpus=[]) and is explicitly instructed never to invent a source id. Zero real sources in, zero real sources out — this is the quantitative baseline Milestone 1 (retrieval) needs to beat, not a bug.richness = 6.0confirms the generator isn't gaming the first two metrics by returning an empty brief.- Mainstream and long-tail are identical here, which means this run cannot yet confirm or deny the project's original hypothesis (that a no-retrieval baseline degrades on long-tail titles) — a judge with 0.0 recall can't see a gap that might exist. See below: this is no longer a missing-dataset problem, it's a judge problem.
The calibration above uses the project's own LLM-written paraphrases —
useful, but it's an LLM checking an LLM. evals/calibrate_substring_external.py
re-runs it against real IMDb user reviews (Misra's IMDB Spoiler Dataset,
Kaggle, free), restricted to the 9 of our 20 titles the dataset covers
(mostly mainstream — long-tail titles here barely have review coverage at
all, a small real echo of the project's own mainstream/long-tail split):
| Value | |
|---|---|
| Reviews evaluated | 2,197 real spoiler-tagged + 5,460 real non-spoiler-tagged |
| Recall | 0.0 — caught 0 of 2,197 |
| Precision | undefined (0 positive predictions made — not "wrong every time") |
Same conclusion, now independently confirmed: SubstringJudge doesn't
just fail on paraphrases it's never seen from itself — it fails on plain
human language. See D12 in docs/DESIGN.md for the one caveat this
comparison carries (a review can be a real spoiler for a plot point we
didn't document, which the method above counts as a miss even though it
isn't the judge's fault).
evals/calibrate_llm_external.py re-runs the exact same method against
LLMJudge (llama-3.1-8b-instant, Groq's free tier) instead of
SubstringJudge. Free-tier limits (1,000 requests/day, and every review
costs len(that movie's labels) calls) meant a smaller, seeded, stratified
sample — 180 reviews (20/title, balanced spoiler/not) instead of the full
7,657 — and review text truncated to 350 characters/call to stay under
the tokens/min cap:
| Judge | n | Recall | Precision |
|---|---|---|---|
SubstringJudge |
7,657 | 0.0 | undefined |
LLMJudge (llama-3.1-8b-instant) |
180 | 0.089 | 0.471 |
A real improvement over the free floor — it sees paraphrases the substring
judge structurally cannot — but not yet trustworthy: it misses ~91 of every
100 real spoiler reveals in this sample, and fewer than half its positive
calls are right. Two things this run can't separate (see D13 in
docs/DESIGN.md): whether that ceiling is the small model or the 350-char
truncation forced by the token budget. Neither judge currently clears the
bar to report a trustworthy leakage_rate.
The decisions behind this design (why there's no LangGraph, why the schema
allows invalid states on purpose, why the same model being measured can't
generate its own ground truth) are documented in
docs/DESIGN.md.
- TMDB — free for non-commercial use, requires attribution (shown in the
app wherever TMDB data appears). Powers the browse tier (
src/preshow/tmdb.py— live search, catalogue posters), separate from the hand-researched, cited-source tier (see D10 indocs/DESIGN.md). Its terms restrict using the content to train AI systems; inference with attribution is the usual reading, but review it before scaling this up further. - OMDb — a path to Rotten Tomatoes/Metascore scores, free tier is limited.
- Wikipedia — CC BY-SA, already in use for researched entries.
- Scraping IMDb — forbidden by ToS, not done under any excuse.
No license defined yet — personal portfolio repo. If you want to reuse something, ask first.