Add dataset sharding + resume on top of qwen-upstream-eval - #13
Open
aysegul-argmax wants to merge 3 commits into
Open
Add dataset sharding + resume on top of qwen-upstream-eval#13aysegul-argmax wants to merge 3 commits into
aysegul-argmax wants to merge 3 commits into
Conversation
The reference-length voice-clone sweep (~718 samples) takes ~26h on one Mac Studio, which no single CI job can hold: GitHub cancels a job at the default 360-minute timeout, and a runner disconnect anywhere in those hours loses the whole run. Nothing in the pipeline could evaluate part of a dataset — only num_samples, which always takes the first N. Add --dataset-shard INDEX/TOTAL so one alias can be swept as several jobs. Shards are interleaved rather than contiguous so that per-sample cost (here reference length, which spans 5s to 550s) spreads evenly instead of piling into the last shard. Membership depends only on INDEX/TOTAL, so a shard that fails can be retried by itself. Two things were needed to make those jobs safe to run against one results repo: - --hf-results-chunk-tag, because the sink resolves its next chunk index once at startup. Concurrent writers would all pick the same index and silently overwrite each other's uploads; distinct tags keep filenames disjoint. - --skip-completed-in, to resume past samples that already have results instead of recomputing them. Ids are read column-only over HTTP, so scanning existing results costs megabytes rather than downloading the embedded audio. Also write one row group per sample, so a reader can range-request a single sample's audio out of a shard instead of pulling the whole file — one long reference clip is already tens of MB. And carry per-sample wsim min / max / min-start in the windowed-SIM detail (keys outside metric_components(), so run totals are untouched), which locates drift in a clip without a separate backfill pass. Co-authored-by: Cursor <cursoragent@cursor.com>
Passing --skip-completed-in put a frozenset of already-scored ids into the dataset config, and wandb serializes that config at init, so every sharded run died with "Object of type frozenset is not JSON serializable" after it had already loaded the dataset and read the existing results. Serialize the id set as a sorted list, and log only how many ids were skipped rather than the ids themselves, which would otherwise swamp the run config. The in-memory value stays a set so per-row membership checks stay O(1). Co-authored-by: Cursor <cursoragent@cursor.com>
Previously `-m sim-windowed` was computed but only SIM/WER landed in parquet, so every sweep still needed a local wSIM backfill. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
berkin/qwen-upstream-eval.--dataset-shard,--skip-completed-in, and--hf-results-chunk-tag.sim-windowedis not enabled).Needed so the 1.7B Python-prototype full sweeps can run as independent shard jobs with resume + failed-shard requeue, instead of one long unsharded job.
Test plan
openbench-cli evaluate --helpshows--dataset-shardand--skip-completed-inshard_spec.total > 1against this pin succeeds--skip-completed-in <results-repo>skips already-scoredsample_idxs