computational-ml-stem-problem-forge

Twelve computationally intensive ML, statistical, and STEM problems. Each one has an unambiguous formalization, a Python reference, an independent verifier, and at least one numerically plausible wrong answer.

A reference solution is not accepted merely because it runs. Each problem states an object, a second calculation of a different invariant, and a third code path. Copying the reference module is not verification.

Entity-intercept leakage from formulation through an incorrect candidate: FLAGSHIP_CASE_STUDY.md. Two of the heavier laboratories (three ground-truth units each) are problems/ml/entity_group_leakage/ and problems/ml/nested_cv_optimism/. Adversarial cases under problems/adversarial/: a leaked AUC that looks like 0.90, and an audit that names the earliest protocol failure. Protocol: docs/ground_truth_protocol.md.

python -m pip install -e .
python -m pytest
python scripts/run_all.py
python -m problemforge list

Python 3.11+. Path-based registry: adding problems/<domain>/<id>/problem.yaml is enough. No hand-maintained index. Twelve problems are complete.

Design

GT1 is the primary object. GT2 is an invariant that is a different computation (split occupancy, majority-classifier identities, translation of log-sum-exp, Gram condition number, Monte Carlo SE). GT3 is a different code path.

Tolerance is a named policy. Coverage is judged by Monte Carlo SE, not phat == 0.95. KKT is a residual, not solver status. Boundaries are tested just inside and just outside.

Leakage has sentinels that do not reuse the primary metric. Entity overlap counts. Source timestamps. Feature permutation while labels stay fixed.

Audit reports the earliest substantive failure. A random split on grouped data is a split failure even if the quoted score is also inner CV.

independent_verifier.py is imported from a qualified path so twelve files named reference_solution.py do not collide. Repo-level tests assert that the two source files are not identical and that a deliberately broken reference fails invariants. SciPy log-sum-exp, Mann–Whitney AUC, analytic cluster-robust variance, NNLS KKT multipliers, and SVD least squares appear as GT3 paths.

Required to keep failing

• Naive KFold on an entity-intercept DGP (entity_group_leakage).
• Inner-CV best score as generalisation (nested_cv_optimism).
• Majority accuracy with recall 0 at ~2% prevalence (imbalance_metrics_threshold).
• A leaked score with AUC in 0.85–0.95 (plausible_wrong_auc).
• iid bootstrap under clustering (bootstrap_dependence_trap).
• (X'X)^{-1}X'y on a Hilbert-like design (ill_conditioned_normal_equations).
• Naive log(sum(exp(x))) overflow (logsumexp_stability).

Locked by per-problem tests and docs/failures_and_corrections.md. Seed regeneration, schema, independence of verifier files, deliberate-failure detection, tolerance boundaries: tests/.

Default seed 2026. Figures and outputs/tables/run_summary.csv are regenerable. There is no observational dataset.

python -m problemforge validate all
python -m problemforge run entity_group_leakage
python -m problemforge audit ai_nested_cv_audit

Corpus

id	difficulty	object
ml/entity_group_leakage	EXPERT	naive vs GroupKFold on intercepts
ml/temporal_feature_leakage	EXPERT	rolling window includes t+1
ml/nested_cv_optimism	EXPERT	inner best vs nested outer
ml/imbalance_metrics_threshold	EXPERT	accuracy vs recall/PR-AUC at 2%
ml/calibration_vs_discrimination	EXPERT	AUC vs ECE
numerical/logsumexp_stability	L4	max-subtraction vs overflow
stats/monte_carlo_ci_coverage	L3	coverage judged by MC SE
stem/qp_kkt_verification	EXPERT	KKT residual, not status
adversarial/plausible_wrong_auc	ADVERSARIAL	leaked 0.9 AUC
adversarial/ai_nested_cv_audit	ADVERSARIAL	earliest protocol failure
stats/bootstrap_dependence_trap	L4	iid vs cluster bootstrap
numerical/ill_conditioned_normal_equations	ADVERSARIAL	Gram inverse vs QR

Limits

The twelve DGPs are stylised. They check procedures. They are not models of a labour market, a clinic, or a trading book. Candidate audit reads declared protocol YAML; it does not parse arbitrary Python. Nested-CV optimism is shown for accuracy on one grid, not for log-loss. The QP laboratory is inequality-only. Passing CI means the laboratory still runs. GT2 must remain a different computation from GT1; gt2 = gt1 would make the independence tests fail. ROADMAP.md.

Pavanam Thomas