Skip to content

feat(val): nested validation dataset groups - #9

Open
AnikethCheluva wants to merge 1 commit into
graph-transform_fixesfrom
graph-nested-val
Open

feat(val): nested validation dataset groups#9
AnikethCheluva wants to merge 1 commit into
graph-transform_fixesfrom
graph-nested-val

Conversation

@AnikethCheluva

Copy link
Copy Markdown

Port of GaTech-RL2#605 onto the graph fork.

valid_datasets may now be {group: {source: dataset}} as well as the
historical {source: dataset}. Lightning runs one val loop per group, in
declaration order, so a run can validate against a held-out split and an
unseen-task split in the same job without conflating their metrics.

The two shapes are told apart by whether the top-level keys are embodiment
names, not by value type -- hydra hands back instantiated objects for the flat
shape and DictConfig for the grouped one, and that distinction is fragile
across omegaconf versions. Note this lookup only disambiguates config shape;
the pipeline runner still treats source keys as opaque and never resolves an
embodiment from them.

A single group returns the bare CombinedLoader, exactly as before, so existing
configs keep their dataloader topology and dataloader_idx stays 0. Two or
more return a list, which Lightning wraps in a sequential CombinedLoader of its
own -- that wrapper passes the inner loader's (batch, batch_idx, dataloader_idx) triple through as the batch, so _unwrap_combined_loader_batch
drops the redundant inner pair before PipelineAlgo sees it.

iter_valid_datasets() is the way to reach all of validation.
valid_datasets is kept as an alias for one group, and iterating it to wire
norm stats is precisely the bug this guards: the skipped groups' datasets stay
unnormalised, the evaluator unnormalises their samples anyway, and the metrics
still look plausible. trainHydra now wires via the iterator and raises if any
dataset is left unwired.

Metrics namespace per group at one choke point: the default group keeps bare
Valid/... names so existing runs overlay on the same charts, every other
group logs Valid_{group}/....

Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01DcPCNAW3GKERANrqvfSCY1

@AnikethCheluva
AnikethCheluva marked this pull request as ready for review September 5, 2026 21:01
@AnikethCheluva
AnikethCheluva changed the base branch from graph-transform_fixes to graphite-base/9 September 8, 2026 23:32
@AnikethCheluva
AnikethCheluva changed the base branch from graphite-base/9 to graph-transform_fixes September 9, 2026 22:21
Port of GaTech-RL2#605 onto the graph fork.

`valid_datasets` may now be `{group: {source: dataset}}` as well as the
historical `{source: dataset}`. Lightning runs one val loop per group, in
declaration order, so a run can validate against a held-out split and an
unseen-task split in the same job without conflating their metrics.

The two shapes are told apart by whether the top-level keys are embodiment
names, not by value type -- hydra hands back instantiated objects for the flat
shape and DictConfig for the grouped one, and that distinction is fragile
across omegaconf versions. Note this lookup only disambiguates config shape;
the pipeline runner still treats source keys as opaque and never resolves an
embodiment from them.

A single group returns the bare CombinedLoader, exactly as before, so existing
configs keep their dataloader topology and `dataloader_idx` stays 0. Two or
more return a list, which Lightning wraps in a sequential CombinedLoader of its
own -- that wrapper passes the inner loader's `(batch, batch_idx,
dataloader_idx)` triple through as the batch, so `_unwrap_combined_loader_batch`
drops the redundant inner pair before PipelineAlgo sees it.

`iter_valid_datasets()` is the way to reach all of validation.
`valid_datasets` is kept as an alias for one group, and iterating it to wire
norm stats is precisely the bug this guards: the skipped groups' datasets stay
unnormalised, the evaluator unnormalises their samples anyway, and the metrics
still look plausible. trainHydra now wires via the iterator and raises if any
dataset is left unwired.

Metrics namespace per group at one choke point: the default group keeps bare
`Valid/...` names so existing runs overlay on the same charts, every other
group logs `Valid_{group}/...`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DcPCNAW3GKERANrqvfSCY1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant