Live site: https://palmer-abs-challenge.vercel.app/
A quant project on the 2026 MLB Automated Ball-Strike (ABS) Challenge System.
Two questions, layered:
- Prediction. Given a called pitch, what's the probability it would be overturned if challenged?
- Policy. Given that probability, the leverage of the moment, and the team's remaining challenge budget, should they challenge?
The headline benchmark: beat MLB's own published "Expected Challenges" model on held-out 2026 regular-season data, and quantify how much run value managers leave on the table by mis-timing their challenges.
| Level | Idea | Holdout Brier | Holdout AUC |
|---|---|---|---|
| L0 | League rate floor | 0.2511 | 0.500 |
| L1 | + rulebook zone distance | 0.1978 | 0.765 |
| L2 | + "reasonable pitch" heuristic | 0.1966 | 0.770 |
| L3 | + empirical-Bayes per-challenger | 0.1832 | 0.798 |
| L4 | + 2-D empirical strike zone | 0.1777 | 0.811 |
| L5 | + pitch characteristics | 0.1769 | 0.817 |
| L6 | GBM ensembled with L5, calibrated | 0.0044 | 1.000 |
ABS is deterministic given precise plate-crossing coordinates. L6 learns that rule; the lower levels show how much of that signal each defensible idea captures on its own. The strategic policy layer is where the work earns its keep — see policy.py.
Training window: 2026-02-21 → 2026-04-30 (975 challenges over 332k pitches). Holdout: 2026-05-01 → 2026-05-15 (209 challenges).
python/
01_eda.py pull one day, profile challenge events
02_fetch_challenges.py full multi-year pitch + challenge pull
02b_fetch_savant_expected.py scrape MLB's own Expected Challenges benchmark
03_features.py per-pitch feature engineering
models.py L0..L6 leaderboard
policy.py backward-induction challenge policy
export_app_data.py package JSON for the web app
data/
pitches_raw.parquet every pitch from the training window
challenges.parquet labeled (overturn yes/no) challenge events
features.parquet feature matrix
predictions.parquet per-pitch model output
validation_metrics.json leaderboard numbers
app_data.json payload the web app loads
app/ Next.js writeup, deployed to Vercel
cd python
pip install -r requirements.txt
python 01_eda.py
python 02_fetch_challenges.py
python 02b_fetch_savant_expected.py
python 03_features.py
python models.py
python policy.py
python export_app_data.py
cd ../app
npm install
npm run dev
Full methodology, level-by-level commentary, limitations, and Tango-formula comparison live in submission.md.