Resume benchmark runs from checkpoints by default - #162
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Benchmark interruptions currently discard the modality runner’s progress even though
BenchmarkRunRecordercan persist rows. This makes checkpointing the default for image, video, and audio runs: rerunning the samerun_idrestores committed predictions and skips those samples before decoding or inference.All modalities continue to use the existing recorder, batch runner, sample IDs, scoring functions, and parquet writer.
BenchmarkRunConfigandBenchmarkPlanbind the model/evaluator/runtime, configuration, seed, source revisions, and selected sample content.DatasetIteratorreplays that frozen selection so cache growth, directory order, and Gasstation week rollover cannot silently change the resumed evaluation. Base and robustness passes retain separate progress; cached augmentation inputs are also frozen. Raw audio and cached tensors share the existing audio loading path, with recorder identity taken from the selected metadata.Each completed inference batch, including model-error rows, is committed before continuing. Checkpoint failures propagate out of dataset/sample handlers. Output parsing completes before any successful rows are appended. Scores and parquet are rebuilt from restored and new rows. Archive-path provenance now follows the same precedence as
build_sample_idfor every recorder status.Storage defaults to
<cache-dir>/runs/<run-id>/checkpoint;--checkpoint-dir/ the Python argument changes its location, with no opt-in flag. An omitted seed resolves to 42. The API forwards the existing persistence callback for filesystems requiring a remote commit. The saved manifest now has its own checksum (format v2), detecting corruption even before the first batch; v1 checkpoints fail closed rather than being reused. PR #161 introduced the storage API but did not enable it in runners.Validation: full unit suite (148 passed, 4 expected skips for audio robustness cases),
ruff check src tests, andgit diff --check. New tests interrupt all three modality loops, cover raw and tensor audio, change/remove cached inputs, resume partial batches and augmentation passes, reject model/input corruption, propagate persistence failures, and compare resumed parquet predictions with uninterrupted runs. A CPU CI job runs the integration tests.Operational boundaries: preparing a new run fingerprints selected source files, adding an initial read pass; pending samples are checked again before decoding. Model setup and uncommitted work may repeat. This PR provides local resume and the durable-filesystem callback; the next bmcore PR will mount persistent storage and replace interrupted Modal sandboxes while retaining the run ID. One coordinator must own each run directory. Submission lifecycle/accounting reconciliation remains a separate follow-up.