Skip to content

New Benchmark: SciKG Verify and SciKG Residual dielectric results - #366

Open
Demfier wants to merge 4 commits into
materialsproject:mainfrom
Demfier:submit-scikg-verify-dielectric
Open

New Benchmark: SciKG Verify and SciKG Residual dielectric results#366
Demfier wants to merge 4 commits into
materialsproject:mainfrom
Demfier:submit-scikg-verify-dielectric

Conversation

@Demfier

@Demfier Demfier commented Sep 2, 2026

Copy link
Copy Markdown

MatBench dielectric submission

This PR adds two named variants for the matbench_dielectric per-task
leaderboard. Both cover all 4,764 candidates in the five official folds.

Model Mean fold MAE Reduction versus MODNet
MODNet v0.1.12 0.2711019242
SciKG Verify 0.2511801222 7.35%
SciKG Residual 0.2472874776 8.78%

Both variants improve every fold. SciKG Verify is the original selector;
SciKG Residual is the pre-existing nonlinear variant, evaluated separately on
all folds. The submission does not mix their predictions according to test
performance. These completed September 6 results replace the earlier archive
in this PR.

Evaluation

For every pair of official folds, MODNet is trained on the other three folds
and predicts both excluded folds. All feature selection, model selection and
training exclude those two folds. Ten such fits supply the correction-training
inputs, so each outer test fold is excluded from their upstream training as
well as from correction fitting. Official outer-fold MODNet predictions remain
unchanged, as do the original correction rules and v2 evidence snapshot.

The verifier uses scientific literature and database evidence published before
2020. This comparison retains the original v2 matching rather than substituting
the later structure-confirmed evidence snapshot. It repairs outer-test training
dependencies while retaining the historical internal selector procedure; it
does not assert that every internal validation layer has been independently
nested. This submission concerns these two MODNet variants only.

Paired bootstrap 95% intervals for the MAE reduction are 0.016048–0.023866
(Verify) and 0.020165–0.027480 (Residual). Resampling is within each fixed fold,
with 20,000 draws; these intervals are conditional on the fitted predictions,
not estimates of retraining variability.

Reproduction

Each benchmark folder contains native results.json.gz, info.json, a
reference snapshot, statistics, a checksum manifest and submission_client.py.

From either folder:

python submission_client.py
python submission_client.py --live
python submission_client.py --live --record reproduced.json.gz

The first command verifies files and recomputes the scores. The second calls
the versioned verifier, which evaluates its fitted correction model on each
request rather than looking up final predictions. The third records those
live outputs through MatBench's API. Requests contain benchmark IDs, fold and
variant, never target labels. No API credential is required.

The public driver covers inference and scoring. Full retraining requires
access to the service-side models and evidence assets; those assets are not
distributed in this PR. The service is scoped to the submitted benchmark IDs.

Both variants have been checked against the live service across all 9,528
predictions with zero numerical difference, and recorded using the MatBench
API. The existing CI compatibility fixes are retained.

Method and release documentation:
https://github.com/Unified-Sciences/scikg-verifier-benchmarks

@Demfier

Demfier commented Sep 2, 2026

Copy link
Copy Markdown
Author

CI note: the benchmark-submission job fails before inspecting this submission. The workflow installs current matminer==0.9.3 / pymatgen==2024.8.9, then requirements-dev.txt downgrades monty to 2022.4.26. Those packages require newer Monty, and import consequently fails with TypeError: deprecated() got an unexpected keyword argument 'deadline' while importing pymatgen. The submission itself is not reached by scripts/test_submission.py.

The added results.json.gz was independently loaded with MatBench 0.6 on the x86 evaluation environment and reports is_valid=True, all five matbench_dielectric folds recorded. Artifact SHA-256: 3c99171a6a140fda4aaee667445c6d4fa6e3ff20c62836962e50f06d49283d75.

@Demfier

Demfier commented Sep 2, 2026

Copy link
Copy Markdown
Author

Maintainer note: the submission guide requires the new_benchmark label. GitHub does not permit this external contributor account to add repository labels, so please add new_benchmark during review. The currently failing checks are upstream CI failures addressed separately in #367.

@Demfier Demfier changed the title Add SciKG Verify MatBench dielectric result New Benchmark: Add SciKG Verify MatBench dielectric result Sep 2, 2026
@Demfier Demfier changed the title New Benchmark: Add SciKG Verify MatBench dielectric result New Benchmark: SciKG Verify and SciKG Residual dielectric results Sep 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant