You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This PR adds two named variants for the matbench_dielectric per-task
leaderboard. Both cover all 4,764 candidates in the five official folds.
Model
Mean fold MAE
Reduction versus MODNet
MODNet v0.1.12
0.2711019242
—
SciKG Verify
0.2511801222
7.35%
SciKG Residual
0.2472874776
8.78%
Both variants improve every fold. SciKG Verify is the original selector;
SciKG Residual is the pre-existing nonlinear variant, evaluated separately on
all folds. The submission does not mix their predictions according to test
performance. These completed September 6 results replace the earlier archive
in this PR.
Evaluation
For every pair of official folds, MODNet is trained on the other three folds
and predicts both excluded folds. All feature selection, model selection and
training exclude those two folds. Ten such fits supply the correction-training
inputs, so each outer test fold is excluded from their upstream training as
well as from correction fitting. Official outer-fold MODNet predictions remain
unchanged, as do the original correction rules and v2 evidence snapshot.
The verifier uses scientific literature and database evidence published before
2020. This comparison retains the original v2 matching rather than substituting
the later structure-confirmed evidence snapshot. It repairs outer-test training
dependencies while retaining the historical internal selector procedure; it
does not assert that every internal validation layer has been independently
nested. This submission concerns these two MODNet variants only.
Paired bootstrap 95% intervals for the MAE reduction are 0.016048–0.023866
(Verify) and 0.020165–0.027480 (Residual). Resampling is within each fixed fold,
with 20,000 draws; these intervals are conditional on the fitted predictions,
not estimates of retraining variability.
Reproduction
Each benchmark folder contains native results.json.gz, info.json, a
reference snapshot, statistics, a checksum manifest and submission_client.py.
The first command verifies files and recomputes the scores. The second calls
the versioned verifier, which evaluates its fitted correction model on each
request rather than looking up final predictions. The third records those
live outputs through MatBench's API. Requests contain benchmark IDs, fold and
variant, never target labels. No API credential is required.
The public driver covers inference and scoring. Full retraining requires
access to the service-side models and evidence assets; those assets are not
distributed in this PR. The service is scoped to the submitted benchmark IDs.
Both variants have been checked against the live service across all 9,528
predictions with zero numerical difference, and recorded using the MatBench
API. The existing CI compatibility fixes are retained.
CI note: the benchmark-submission job fails before inspecting this submission. The workflow installs current matminer==0.9.3 / pymatgen==2024.8.9, then requirements-dev.txt downgrades monty to 2022.4.26. Those packages require newer Monty, and import consequently fails with TypeError: deprecated() got an unexpected keyword argument 'deadline' while importing pymatgen. The submission itself is not reached by scripts/test_submission.py.
The added results.json.gz was independently loaded with MatBench 0.6 on the x86 evaluation environment and reports is_valid=True, all five matbench_dielectric folds recorded. Artifact SHA-256: 3c99171a6a140fda4aaee667445c6d4fa6e3ff20c62836962e50f06d49283d75.
Maintainer note: the submission guide requires the new_benchmark label. GitHub does not permit this external contributor account to add repository labels, so please add new_benchmark during review. The currently failing checks are upstream CI failures addressed separately in #367.
Demfier
changed the title
New Benchmark: Add SciKG Verify MatBench dielectric result
New Benchmark: SciKG Verify and SciKG Residual dielectric results
Sep 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MatBench dielectric submission
This PR adds two named variants for the
matbench_dielectricper-taskleaderboard. Both cover all 4,764 candidates in the five official folds.
Both variants improve every fold. SciKG Verify is the original selector;
SciKG Residual is the pre-existing nonlinear variant, evaluated separately on
all folds. The submission does not mix their predictions according to test
performance. These completed September 6 results replace the earlier archive
in this PR.
Evaluation
For every pair of official folds, MODNet is trained on the other three folds
and predicts both excluded folds. All feature selection, model selection and
training exclude those two folds. Ten such fits supply the correction-training
inputs, so each outer test fold is excluded from their upstream training as
well as from correction fitting. Official outer-fold MODNet predictions remain
unchanged, as do the original correction rules and v2 evidence snapshot.
The verifier uses scientific literature and database evidence published before
2020. This comparison retains the original v2 matching rather than substituting
the later structure-confirmed evidence snapshot. It repairs outer-test training
dependencies while retaining the historical internal selector procedure; it
does not assert that every internal validation layer has been independently
nested. This submission concerns these two MODNet variants only.
Paired bootstrap 95% intervals for the MAE reduction are 0.016048–0.023866
(Verify) and 0.020165–0.027480 (Residual). Resampling is within each fixed fold,
with 20,000 draws; these intervals are conditional on the fitted predictions,
not estimates of retraining variability.
Reproduction
Each benchmark folder contains native
results.json.gz,info.json, areference snapshot, statistics, a checksum manifest and
submission_client.py.From either folder:
The first command verifies files and recomputes the scores. The second calls
the versioned verifier, which evaluates its fitted correction model on each
request rather than looking up final predictions. The third records those
live outputs through MatBench's API. Requests contain benchmark IDs, fold and
variant, never target labels. No API credential is required.
The public driver covers inference and scoring. Full retraining requires
access to the service-side models and evidence assets; those assets are not
distributed in this PR. The service is scoped to the submitted benchmark IDs.
Both variants have been checked against the live service across all 9,528
predictions with zero numerical difference, and recorded using the MatBench
API. The existing CI compatibility fixes are retained.
Method and release documentation:
https://github.com/Unified-Sciences/scikg-verifier-benchmarks