Evaluation pipeline for the Falcon Language Support project. Evaluates regional LLMs across translation and instruction-following tasks in 30+ languages, using an LLM-as-judge pointwise scoring methodology, and grades each regional model A–D on absolute output quality.
Runs entirely on Modal — a FastAPI model registry, GPU inference workers, and a judge worker, wired together by one state machine, plus a new automation layer that chains tokenizer testing through to a live dashboard deploy for adding new challenger models.
For current results and project status, see docs/status.md.
For detailed architecture notes, see docs/architecture.md.
For how to run things, see docs/runbook.md.
- Architecture
- The pipeline state machine (v2)
- Automated rollout pipeline
- Grading
- Setup
- Running the pipeline
- Registry API
- Repository layout
- Testing
Two subsystems share one Modal app and one set of volumes:
┌─────────────────────────────────────────────────────────────┐
│ Modal app (modal_app.py) │
│ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Registry service — hexagonal architecture, FastAPI │ │
│ │ │ │
│ │ api/routes.py → RegistryService → RegistryStore │ │
│ │ (port: Protocol) │ │
│ │ ├─ VolumeRegistryStore (production) │ │
│ │ └─ InMemoryRegistryStore (tests) │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ ─────────── shared Modal volumes ────────── │
│ registry · weights · benchmarks · outputs │
│ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Evaluation pipeline — one state machine (v2) │ │
│ │ │ │
│ │ run_pipeline() / rollout() │ │
│ │ → state_machine.advance() │ │
│ │ PENDING → SAMPLE → INFERENCE → JUDGE │ │
│ │ → REPORT → DONE │ │
│ │ ↘ FAILED (from any state) │ │
│ │ │ │
│ │ Workers: VLLMWorker · JudgeWorker · ReportGenerator│ │
│ └─────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
Registry service. RegistryStore is a typing.Protocol, not a base class —
the service layer (RegistryService) depends only on the protocol, never on a
concrete adapter. Swapping VolumeRegistryStore for InMemoryRegistryStore is
one environment variable (PERSISTENCE_BACKEND=memory|volume). The test suite
forces memory so every test runs against a fresh in-memory store with zero
Modal dependency. Domain objects (ModelConfig, HardwareConfig, MetricConfig,
TaskConfig) are plain dataclasses with no framework imports. Inference
workers do NOT call this service — they read /data/registry/models.json
directly off the mounted volume, avoiding a network round-trip per container.
Evaluation pipeline. There is one state machine, not three separate
orchestrators. run_pipeline() takes a list of LanguageSpecs (1 or many) and
a stop_at state, and drives every spec through advance(). Running 1
language vs many only changes whether the work happens inline in the current
process or fanned out to parallel Modal containers.
PENDING → SAMPLE → INFERENCE → JUDGE → REPORT → DONE
↘ FAILED
| State | What happens | Worker |
|---|---|---|
SAMPLE |
Stratified random sample (200 by default) drawn from the full benchmark set — by direction for translation, category for instructions |
(inline, CPU) |
INFERENCE |
Runs the regional model on the sampled subset. generate() for one language, or generate_multi() for several languages sharing one GPU container (see automated rollout) |
VLLMWorker |
JUDGE |
Pointwise — Gemini/Claude scores the regional model's output standalone, 0–1, per rubric dimension. No Gemma-4 comparison | JudgeWorker |
REPORT |
Aggregates verdicts into final_report.json, computes avg_score and an A–D grade |
ReportGenerator |
Any state can transition to FAILED; the error is recorded on the
LanguageRun and surfaced in the final summary rather than crashing the run.
Changed from v1: no Gemma-4 baseline step, no LIGHT_METRICS/MODEL_METRICS
states (automated BLEU/chrF/COMET/BERTScore metric classes still exist in
src/metrics/ but are no longer wired into the v2 pipeline — grading is
judge-only). Judging is pointwise, not pairwise, so there's no swap-run/position-bias
handling either.
# Single language, full pipeline through judging + report
modal run modal_app.py::run_pipeline --slug greek
# A few languages in parallel
modal run modal_app.py::run_pipeline --slug arabic,hebrew,amharic
# All registered models — prints a cost estimate, confirms before launching
modal run modal_app.py::run_pipeline --slug all
# Resume an existing run — already-completed languages are skipped
modal run modal_app.py::run_pipeline --slug all --run-id <existing-run-id>
# Re-judge existing GPU outputs with the current rubric, no new inference
modal run modal_app.py::run_pipeline --slug german --task translation \
--rejudge-from <old-run-id>| Flag | Default | Meaning |
|---|---|---|
--slug |
all |
one slug, comma-separated list, or all |
--model-id |
(none) | filter by model_id, comma-separated |
--stop-at |
report |
sample, inference, judge, or report |
--task |
translation |
translation or instructions |
--n-samples |
200 |
stratified sample size |
--seed |
42 |
sampler random seed |
--judge-model |
gemini-3.5-flash |
gemini-* or claude-* |
--rejudge-from |
(none) | skip inference, reuse an old run's outputs, judge only |
--run-id |
(generated) | resume an existing run |
Adding a new challenger model no longer requires running each stage by hand.
scripts/rollout.py + modal_app.py::rollout chain: tokenizer test →
shortlist gate (human-confirmed, writes config/model_rollouts/<slug>.yaml)
→ pretranslate → shared-batch GPU inference → per-language judge/report →
per-language dashboard publish.
Shared-batch inference is the key efficiency change: instead of one GPU
container per language (each reloading the same model weights), a chunk of
languages is processed in ONE container via VLLMWorker.generate_multi().
The existing per-language JUDGE→REPORT fan-out then runs unmodified — SAMPLE
and INFERENCE auto-skip since the shared-batch step already populated their
outputs.
Per-language incremental publish: as each language's judge+report resolves
(not once at the end of a big batch), scripts/publish.py::publish_run()
updates the dashboard JSON, generates review HTML, commits, and deploys — so
results appear as they complete, not all at once.
python scripts/rollout.py tokenizer-test --model EuroLLM-22B
python scripts/rollout.py shortlist --model EuroLLM-22B # writes + stops for human review
modal run --detach modal_app.py::rollout --model eurollm-22b --no-deploy # smoke test
modal run --detach modal_app.py::rollout --model eurollm-22b # for realSee docs/runbook.md for the full command reference.
Each language's final_report.json entry gets a grade from avg_score (mean
across all judged rubric dimensions):
| Grade | avg_score |
|---|---|
| A | ≥ 0.75 |
| B | ≥ 0.50 |
| C | ≥ 0.25 |
| D | < 0.25 |
Only A and B grades move forward to the next phase (Whisper fine-tuning). See
docs/status.md for the current per-language grade table.
- A Modal account,
modalCLI installed and authenticated - A Gemini API key (paid tier recommended for full 200-sample runs — free tier is rate-limited to ~15 requests/minute)
modal secret create phase2a-registry-url REGISTRY_URL=<your-registry-url> JWT_TOKEN=<token>
modal secret create phase2a-auth-secrets JWT_SECRET=<32+ byte secret> HF_TOKEN=<huggingface-token>
modal secret create phase2a-judge GEMINI_API_KEY=<your-key>phase2a-judge can also carry ANTHROPIC_API_KEY for a claude-* judge model.
Created automatically on first use:
| Volume | Mount | Contents |
|---|---|---|
phase2a-registry |
/data/registry |
model/hardware/metric/task configs |
phase2a-weights |
/data/weights |
HuggingFace model cache |
phase2a-benchmarks |
/data/benchmarks |
FLORES-200 + instruction-following datasets |
phase2a-outputs |
/data/outputs |
inference outputs, judge verdicts, reports |
pip install -r requirements.txt # registry service: fastapi, uvicorn
pip install pytest pyjwt httpx pydantic sacrebleu langdetect # test suite extras
pip install fastapi uvicorn pyyaml # scripts/dashboard.py, scripts/rollout.py, scripts/publish.py
PERSISTENCE_BACKEND=memory pytest tests/ -qThe eval pipeline itself (vLLM inference, judge calls) runs entirely inside Modal containers — nothing GPU-related needs installing locally.
modal deploy modal_app.pyDeploys the registry service and exposes its URL. Note: the evaluation
pipeline and rollout entrypoints are run ephemerally via modal run, not
modal deploy — only the RegistryService needs to be a persistent deployment.
# 1. Generate a write-scoped JWT for the registry
export JWT_TOKEN=$(python scripts/jwt_token_generator.py --secret $JWT_SECRET --scopes registry:write)
export REGISTRY_URL=https://<your-deployed-registry-url>.modal.run
# 2. Seed the registry with model configs + hardware + metrics + tasks
python scripts/seed_registry.py
# 3. Fetch chat templates for instruct-variant models
python scripts/fetch_chat_templates.py
# 4. Try one language first
echo "" | modal run --detach modal_app.py::run_pipeline --slug greek --n-samples 20
# 5. Once that looks right, a small subset
modal run modal_app.py::run_pipeline --slug arabic,hebrew,amharic
# 6. Or use the automated rollout for a new model across many languages
# (see "Automated rollout pipeline" above)Outputs land in /data/outputs/runs/{run_id}/:
runs/{run_id}/
sampled/{slug}_{task}_ids.json
regional/{slug}_{task}_outputs.jsonl
judge/{slug}_{task}_verdicts.jsonl
reports/
final_report.json # all languages run so far, this run_id
{slug}_summary.md # single-language run
run_summary.md # multi-language run
run_manifest.json
FastAPI service deployed via Modal, exposing:
| Resource | Endpoints |
|---|---|
| Models | GET/POST /models, GET/PATCH /models/{id}, PATCH /models/{id}/{enable,disable,deprecate}, POST /models/{id}/fetch-chat-template |
| Hardware | GET/POST /hardware, GET/PATCH /hardware/{id} |
| Metrics | GET/POST /metrics, GET/PATCH /metrics/{name} |
| Tasks | GET /tasks, GET/PATCH /tasks/{name} |
| Runs | GET/POST /runs, GET /runs/{id} |
| Health | GET /health |
All mutating endpoints require a JWT bearer token with the appropriate scope
(registry:read / registry:write). Model state transitions (active →
disabled/deprecated) are validated by src/core/services/state_machine.py
— a separate, smaller state machine from the evaluation pipeline's;
deprecated is terminal.
modal_app.py Modal app — worker registrations, run_pipeline/rollout entrypoints
modal_common.py Volumes, images, GPU presets
src/
main.py FastAPI app
deps.py PERSISTENCE_BACKEND dependency wiring
core/
domain/ ModelConfig, HardwareConfig, MetricConfig, TaskConfig
ports/ RegistryStore protocol
services/
registry_service.py business logic, depends only on the port
state_machine.py model lifecycle transitions (active/disabled/deprecated)
adapters/persistence/
volume_store.py production: Modal volume-backed
memory_store.py tests: in-memory
api/
routes.py all registry endpoints
auth.py JWT verification
schemas.py Pydantic request/response models
pipeline/
state_machine.py the v2 evaluation pipeline state machine
entrypoints.py run_pipeline(), cost estimation
sampler.py stratified sampling
loader.py benchmark sample loading, prompt building
run.py path helpers, manifest I/O
workers/
inference.py VLLMWorker — generate(), generate_multi() (no src/ imports, see below)
judge.py JudgeWorker — pointwise Gemini/Anthropic scoring
reporter.py ReportGenerator, grading logic
metrics/ Legacy v1 automated metric classes (BLEU/chrF/etc.) —
not wired into the v2 pipeline, kept for reference
scripts/
dashboard.py FastAPI dashboard app (local + Vercel)
publish.py publish_run() — dashboard entry + review HTML + deploy
rollout.py tokenizer-test / shortlist CLI subcommands
generate_review.py per-run review HTML generator
seed_registry.py populates model configs
fetch_chat_templates.py pulls chat templates for instruct models
jwt_token_generator.py generates JWTs for manual API testing
tests/ registry CRUD, auth/JWT scopes, model state
transitions, path helpers, manifest I/O
src/workers/inference.py deliberately does not import anything from the rest
of src/ — the vLLM Modal image doesn't have the registry package installed,
so it re-declares the small subset of ModelConfig fields it needs as a local
_ModelInfo dataclass and reads /data/registry/models.json directly.
PERSISTENCE_BACKEND=memory pytest tests/ -qCovers registry CRUD, auth/JWT scopes, model state transitions, path helpers, manifest I/O, and the evaluation state machine. Judge/inference logic isn't covered by the test suite — it requires real API calls / GPU and is exercised manually against live runs instead.