Skip to content

Resume benchmark runs from checkpoints by default - #162

Merged
dylanuys merged 2 commits into
devfrom
feat/benchmark-checkpoint-resume
Sep 21, 2026
Merged

dylanuys merged 2 commits into
devfrom
feat/benchmark-checkpoint-resume

Conversation

@dylanuys

Copy link
Copy Markdown
Contributor

Benchmark interruptions currently discard the modality runner’s progress even though BenchmarkRunRecorder can persist rows. This makes checkpointing the default for image, video, and audio runs: rerunning the same run_id restores committed predictions and skips those samples before decoding or inference.

All modalities continue to use the existing recorder, batch runner, sample IDs, scoring functions, and parquet writer. BenchmarkRunConfig and BenchmarkPlan bind the model/evaluator/runtime, configuration, seed, source revisions, and selected sample content. DatasetIterator replays that frozen selection so cache growth, directory order, and Gasstation week rollover cannot silently change the resumed evaluation. Base and robustness passes retain separate progress; cached augmentation inputs are also frozen. Raw audio and cached tensors share the existing audio loading path, with recorder identity taken from the selected metadata.

Each completed inference batch, including model-error rows, is committed before continuing. Checkpoint failures propagate out of dataset/sample handlers. Output parsing completes before any successful rows are appended. Scores and parquet are rebuilt from restored and new rows. Archive-path provenance now follows the same precedence as build_sample_id for every recorder status.

Storage defaults to <cache-dir>/runs/<run-id>/checkpoint; --checkpoint-dir / the Python argument changes its location, with no opt-in flag. An omitted seed resolves to 42. The API forwards the existing persistence callback for filesystems requiring a remote commit. The saved manifest now has its own checksum (format v2), detecting corruption even before the first batch; v1 checkpoints fail closed rather than being reused. PR #161 introduced the storage API but did not enable it in runners.

Validation: full unit suite (148 passed, 4 expected skips for audio robustness cases), ruff check src tests, and git diff --check. New tests interrupt all three modality loops, cover raw and tensor audio, change/remove cached inputs, resume partial batches and augmentation passes, reject model/input corruption, propagate persistence failures, and compare resumed parquet predictions with uninterrupted runs. A CPU CI job runs the integration tests.

Operational boundaries: preparing a new run fingerprints selected source files, adding an initial read pass; pending samples are checked again before decoding. Model setup and uncommitted work may repeat. This PR provides local resume and the durable-filesystem callback; the next bmcore PR will mount persistent storage and replace interrupted Modal sandboxes while retaining the run ID. One coordinator must own each run directory. Submission lifecycle/accounting reconciliation remains a separate follow-up.

@dylanuys
dylanuys merged commit 5521899 into dev Sep 21, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant