An open, engine-agnostic benchmarking and regression-testing platform for SWMM-compatible engines. 1,396 models — EPA and OWA regression examples, the EXTRAN manual problems, the EPA QA suite with its original SWMM4 references, analytical test problems with exact solutions, and real-world networks — run automatically against multiple engines and compared along dimensions that matter to practitioners.
Status: under construction. The harness is scaffolded and the corpus is migrated (1,396 tagged, schema-valid cases). Suite migrations, the published dashboard, and the engine-repo switchover are still to come.
No engine has been run through the harness yet — everything verifiable without one is verified, and the rest is not. See the agent handoff for exactly what is and is not proven, and the platform plan for the design and implementation order.
| dimension | what it answers |
|---|---|
| Stability | Does the solver converge? Does the timestep collapse? Does it crash? |
| Mass balance | Runoff, routing, and quality continuity error |
| Result parity | Every subcatchment, node, link, and system variable — including pollutants — at every reported timestep |
| Analytical accuracy | L1/L2/L∞ error norms and observed convergence order against exact solutions |
| Performance | Wall-clock per model per engine, tracked historically |
Only cases with a genuine exact solution (reference.class: analytic or
manufactured) can show an engine is wrong; the rest can only show that
engines differ. The platform keeps these apart deliberately — error norms
are computed for truth-class cases alone, and the verification and
regression badges are separate numbers. "1,396 models pass" is a regression
claim, not an accuracy claim, and is never presented as one.
pip install -r requirements.txt
python -m pytest # harness tests (no engine needed)
python run_regression.py --list # available suites
python run_regression.py --suite parity --tier pr # the PR-tier sweep
python run_regression.py --suite all # everything + reports
python -m harness.engines # what resolved, and where
python -m harness.corpus --census # models per tag
python -m harness.corpus "lid AND pollutants" # tag query
python -m harness.validate # validate the corpusEngines are located through environment variables — OPENSWMM_ENGINE_DIR,
OPENSWMM_BUILD_DIR, OPENSWMM_EXE, OPENSWMM_LEGACY_EXE — so the same
command runs locally and in CI. A missing engine degrades to UNAVAILABLE,
never a crash.
harness/ engines · runner · readers · rptparse · compare · scoring · corpus · validate
engines.yaml THE engine registry — engines are data, not code
schemas/ metadata + provenance schemas, tag vocabulary
suites/ parity · epa_qa · analytical/{swashes,transitions,manufactured}
· stability · quality · performance
corpus/ the model library: <collection>/<case>/{model.inp,metadata.yaml,provenance.yaml}
data/ shared forcing files (rainfall, timeseries)
legacy/ pre-migration source material (XPSWMM projects, summaries)
site/ dashboard templates (generated output is never committed)
plans/ design documents
Engines are declared in harness/engines.yaml, not in
code. Three source types: in-tree (a local build), git-ref (built from a
ref, cached), and external-binary (any SWMM-compatible CLI). External engines
are report-only — never gating — and are excluded from cross-engine
performance claims unless their build configuration is asserted comparable.
See CONTRIBUTING.md. Models are classified by multiple tags, carry provenance and a license, and can be run through an anonymization tool if the network can't be shared as-is.
The Unlicense (see LICENSE), matching EPA SWMM5's public-domain
status. Individual cases declare their own license in provenance.yaml;
anything marked unverified is excluded from redistribution claims. Three
collections — greenville, simon-epa, special — are contributed or
special-case models currently marked unverified pending provenance
confirmation.