Skip to content

Seed novel's internal validation split - #46

Open
rcannood wants to merge 2 commits into
mainfrom
fix/novel-unseeded-batch-fallback
Open

Seed novel's internal validation split#46
rcannood wants to merge 2 commits into
mainfrom
fix/novel-unseeded-batch-fallback

Conversation

@rcannood

Copy link
Copy Markdown
Member

Describe your changes

novel_train reproduces the competition's phase-1 split by name, with a fallback:

test_batches = {'s1d2', 's3d7'}
if len(test_batches.intersection(set(batch))) == 0:
  all_batches = batch.cat.categories.tolist()
  test_batches = set(np.random.choice(all_batches, math.ceil(len(all_batches) * 0.25), replace=False))

I'd assumed the fallback was dormant. It isn't -- bmmc_multiome doesn't contain either batch:

bmmc_cite/normal       batches=['s1d1','s1d2','s1d3','s2d1','s2d4','s2d5','s3d1','s3d6','s3d7']  hardcoded present=['s1d2','s3d7']
bmmc_multiome/normal   batches=['s1d1','s2d1','s2d4','s3d6']                                     hardcoded present=NONE

So on GEX2ATAC and ATAC2GEX -- half our datasets -- the validation batch is drawn unseeded:

unseeded np.random.choice (current):
    ['s1d1']
    ['s3d6']
    ['s3d6']
seeded default_rng(1) (new):
    ['s2d1']
    ['s2d1']
    ['s2d1']

That's the split train_and_valid() uses to decide which checkpoint to keep, so which model novel ships changes run to run, and there's no way to pin it. Added a --seed argument (default 1) and a line logging the chosen batches, since "which batches were held out" is otherwise invisible in the logs.

Deliberately not touching the hardcoded {'s1d2', 's3d7'} -- that's faithful to the original submission and does the right thing on CITE.

Note this changes which batch is held out on the multiome datasets (whatever the RNG happened to give, to s2d1), so novel's multiome scores will shift slightly. That seemed better than leaving them unpinnable.

Part of a series of PRs coming out of a pre-run review of the benchmark.

Checklist before requesting a review

  • I have performed a self-review of my code

  • Check the correct box. Does this PR contain:

    • Breaking changes
    • New functionality
    • Major changes
    • Minor changes
    • Bug fixes
  • Proposed changes are described in the CHANGELOG.md

  • CI Tests succeed and look good!

rcannood added 2 commits July 29, 2026 14:12
The competition batches s1d2/s3d7 aren't in bmmc_multiome, so both
multiome datasets fall through to an unseeded np.random.choice and pick
different validation batches on every run. Add --seed and log the pick.
@rcannood

Copy link
Copy Markdown
Member Author

The failing novel_train test here isn't from this change -- --output never reaches the command line because file_pretrained_model.yaml has no example, so run_and_check_output skips it. #33 hit the same failure before it was merged. Fixed separately in #52; this should go green once that lands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant