fix(datasets): backfill .static on caches predating it (PF test/predict) - #125
Open
romeokienzler wants to merge 1 commit into
Open
romeokienzler wants to merge 1 commit into
romeokienzler wants to merge 1 commit into
Conversation
RemovePFMask (PF test_step/predict_step) reads `.static` limit snapshots that build_hetero_data attaches at graph-build time. Datasets processed before `.static` was introduced (PRs #99/#105/#122) carry the limit columns only inside `bus.x` / branch `edge_attr`, so loading such a cache and running the PF test/predict path crashes with: AttributeError: 'NodeStorage' object has no attribute 'static' process() skips regeneration when a `processed_raw_files.done` marker exists, so stale caches are never rebuilt. Add backfill_static(), invoked in HeteroGridDatasetDisk.get() right after from_dict and before normalisation, which reconstructs `.static` from the raw (pre-mask, pre-normalisation) limit columns — identical to what build_hetero_data stores. Freshly processed graphs already carry `.static`, so the backfill is a no-op for them. Also centralise the limit-column indices as BUS_STATIC_COLS / BRANCH_STATIC_COLS so build_hetero_data and backfill_static share one source of truth. Verified on vela (torch 2.13.0, H100): integrationtests PF + OPF both pass against the stale case14 fixture (2 passed, 0:45:40); no AttributeError. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Romeo Kienzler <romeo.kienzler1@ibm.com>
Collaborator
Author
|
@romeokienzler thanks for the fix — the diagnosis is clear and the What's needed
Otherwise this looks tightly scoped and forward-compatible with no reprocessing required, which is the right call. Deferring the merge decision to a maintainer. — 🤖 _automated pre-review; a maintainer will follow up_ |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The PF test/predict path crashes on any dataset processed before
.staticwas introduced:RemovePFMask(intest_step/predict_step) restores static limit columns from the.staticsnapshot that
build_hetero_dataattaches at graph-build time. But.staticwas added after manydatasets were already processed to disk.
HeteroGridDatasetDisk.process()skips regeneration when aprocessed_raw_files.donemarker exists, so those stale caches are never rebuilt — their graphs carrythe limit columns only inside
bus.x/ branchedge_attr, and loading one crashes the PFtest/predict path.
This was surfaced by running the training integration tests (
integrationtests/test_base_set.py::test_train_pf)on
main: the committed case14 fixture ships pre-processeddata_index_*.ptgraphs built before.staticexisted, so every PF run aborts in the test phase. (OPF is unaffected — it doesn't use.static.)Fix
backfill_static(data)indatasets/graph_builder.py: if a graph lacks.static, reconstruct itfrom the raw limit columns still present in
bus.x/ branchedge_attr— identical to whatbuild_hetero_datastores. Freshly-processed graphs already carry.static, so it's a no-op for them.HeteroGridDatasetDisk.get()right afterHeteroData.from_dictand beforenormalisation, so the captured snapshot matches build-time raw semantics (and branch masking slices
it consistently afterward).
BUS_STATIC_COLS/BRANCH_STATIC_COLSso the build path andthe backfill path share one source of truth.
No dataset regeneration required — stale caches become forward-compatible on load.
Validation
On vela (torch 2.13.0+cu130, H100),
pytest integrationtestsagainst the stale case14 fixture:2 passed in 0:45:40(PF + OPF), noAttributeError. Before the fix, PF crashed in the test phase.🤖 Generated with Claude Code