Small justified subsets of learners on documented synthetic DGPs. I keep the invalid workflow's number next to the scientifically matched number. Those draws are method checks, not empirical findings. Perfect ROC-AUC can coexist with worse Brier score and log loss.
Three error locations (preprocessor, splitter, inner best_score_) that manufacture a headline: CASE_STUDY_MODEL_SELECTION_FAILURE.md. The ten designed failures and the tests that keep them visible: docs/failures_and_corrections.md. Named leaks and named estimands: docs/leakage_taxonomy.md, docs/validation_design.md.
Each experiment module under src/mslab/experiments/ returns naive_score, correct_score, diagnostics, and a what_failed paragraph. A catalogue exists in src/mslab/models/catalogue.py; JUSTIFIED_SUBSETS forbids running every model on every DGP.
python -m pip install -e .
python -m pytest
python scripts/run_all.pyPython 3.11+. Printed numbers come from that script. They are not hard-coded here.
Every splitter docstring in validation/designs.py names the risk it estimates. GroupKFold is not “more rigorous KFold”; it is a different sampling unit.
The invalid preprocessor is fit_preprocessor_on_full_frame. The valid one is fit_preprocessor_on_train. Tests compare scaler means.
nested_cv_evaluate returns the outer mean. Inner best_score_ is a diagnostic.
Accuracy, ROC-AUC, PR-AUC, Brier, and ECE answer different questions. Majority accuracy on prevalence 5% is a closed-form trap: expected accuracy = 0.95, recall = 0.
Permutation importance is not causal. I use it, and I say so in docs/model_interpretation_limits.md. SHAP is not a dependency.
Justified subsets: logistic for leakage and snooping; a tree grid for search optimism; a shallow tree for covariate shift (local leaves, source support); a forest for impurity ranks and overconfident probabilities; a dummy next to a forest when there is no signal.
DGPs: informative vs noise, correlated proxy, grouped intercepts, rare events, time plus a future sentinel, covariate shift. No observational extract.
- Supervised selection fitted on all rows inflates holdout AUC.
- Shuffled KFold with entity codes overstates risk for a new entity.
- A post-outcome sentinel plus a shuffled split overstates forward risk.
GridSearchCV.best_score_sits above nested outer accuracy.- Majority accuracy is high; rare-class recall is zero.
- Full-sample correlation screening scored on the same rows overstates AUC.
- Impurity ranks a correlated proxy that has structural coefficient 0.
- Forest ROC-AUC looks strong while ECE does not.
- Source holdout is not target risk under covariate shift.
- Forest training accuracy on labels independent of x is memorisation.
Property checks in tests/: holdout indices partition {0,…,n−1}; TimeSeriesSplit trains only on earlier times and the future sentinel is dropped on the correct path; GroupKFold never shares an entity; leaky scaler means match a full-frame fit; nested CV outer test ∩ inner train = ∅; on noise_features_dgp, mean permutation importance of informative columns exceeds noise; true Bernoulli probabilities have small ECE; seed 2026 reproduces experiment scores; majority classifier accuracy ≥ 0.90 and recall = 0 on the 5% DGP.
scripts/run_all.py writes outputs/figures/ and outputs/tables/run_summary.csv. Regenerable.
Stylised DGPs — tools for checking procedures, not models of a clinic or a credit book. A score gap on a designed DGP is not a meta-analysis of papers. Nested CV estimates the risk of a selection procedure under the splitter's sampling model; it is not proved unbiased here. Permutation importance, impurity, and coefficients are not causal effects. No deployed-system or latency claim.
Not implemented: purged/embargoed splits beyond TimeSeriesSplit(gap=...); importance-weighted target-risk under covariate shift. ROADMAP.md.
There is no observational dataset. Provenance: docs/data_policy.md. Causal identification, when that is the question: econometrics-causal-inference-lab.
Pavanam Thomas