Add SAL precipitation figures (S–A scatter and case-snapshot maps) to the publication workflow - #223
Draft
Louis-Frey wants to merge 62 commits into
Draft
Add SAL precipitation figures (S–A scatter and case-snapshot maps) to the publication workflow#223Louis-Frey wants to merge 62 commits into
Louis-Frey wants to merge 62 commits into
Conversation
AI-assisted implementation!
This PR adds meteograms for publication, centralises some ploting style
things, uses long names for variables and tries out some colors :)
Any feedback and changes welcome!
To run only plotting:
```
evalml publication config/Varda-Single_paper.yaml -- \
--allowed-rules publication_all publication_meteogram publication_figures \
data_download_eckit_geo_grids \
--forcerun publication_meteogram publication_figures
```
Current version of the plots:
<img width="2957" height="1685" alt="image"
src="https://github.com/user-attachments/assets/218f849e-4bd3-41d2-b19d-124dfe156e50"
/>
<img width="2335" height="2056" alt="image"
src="https://github.com/user-attachments/assets/25d1d544-996a-4adf-869c-54ae054cfcf8"
/>
---------
Co-authored-by: Jonas Bhend <jonas.bhend@meteoswiss.ch>
Co-authored-by: Daniele Nerini <daniele.nerini@meteoswiss.ch>
Co-authored-by: Michele Cattaneo <44707621+MicheleCattaneo@users.noreply.github.com>
Co-authored-by: Jonas Bhend <jonasbhend@users.noreply.github.com>
) AI assisted implementation! - Derive the jretrievedwh obs marker from --station inside the script instead of passing a separate --obs argument. The marker value contains '=' (e.g. jretrievedwh:locations=KLO), which marimo's cli_args drops, so obs silently fell back to the KLO default and any non-KLO station was broken (also via the snakemake rule). The rule no longer passes --obs. - Import `time` in the entry cell and thread it through: the timing log lines used time.perf_counter() but `time` was never imported (NameError at runtime).
AI assisted implementation! Changes the plotting workflow for publication to be able to generate them from the snakemake workflow as well as interactively rendering through a manifest . A documentation is currently available in `docs/publication_figures.md`
This PR adds a standalone script that plots an areal-mean meteogram over a region polygon, styled to match the paper figures. It masks the truth points inside a region shapefile, averages over them at each valid time, and plots the series, reusing `publication_style` (`line_style`, `param_label`), `publication.mplstyle`, and the lead-time x-axis of `publication_meteogram` so it reads consistently with the other paper figures. The script is standalone (not wired to snakemake) and truth-only: it reads the analysis zarr directly, with no inference or verification. It works for any region shapefile and parameter. For the Valais precipitation case study it plots the areal-mean hourly precipitation of the ICON-CH1 analysis over the Valais region. The Valais shapefile is on store at `/store_new/mch/msopr/ml/regions/cantons/valais.shp`. It can also be generated with the included `make_region_shapefile.py`, which pulls the canton boundary from GADM and reprojects it to EPSG:2056: ``` python workflow/scripts/make_region_shapefile.py --canton Valais --outfn valais.shp ``` ## Output With ``` python workflow/scripts/plot_meteogram_region.py \ --truth /store_new/mch/msopr/ml/datasets/mch-ich1-1km-2024-2025-1h-pl13-v1.0.zarr \ --truth_label "ICON-CH1 analysis" \ --shapefile /store_new/mch/msopr/ml/regions/cantons/valais.shp \ --date 202506271800 --steps 0/120/1 --param TOT_PREC --outfn output/results/valais.png ``` you get: <img width="1560" height="585" alt="valais_final" src="https://github.com/user-attachments/assets/779cef33-3894-473b-8d6f-890379f0659a" />
SAL (Structure–Amplitude–Location) precipitation scoring uses pysteps.verification.salscores; scikit-image is its transitive requirement. uv.lock is gitignored, so no lockfile update.
Framework-agnostic helpers for Structure–Amplitude–Location scoring: build a near-isotropic regular lat–lon raster, nearest-neighbour remap native (triangular ICON / KENDA) fields onto it via verification.spatial.spherical_nearest_neighbor_indices, and a thin wrapper over pysteps' sal() that gates dry windows. S/A/L are normalized ratios, hence invariant to a constant precipitation rescaling; the raster is required because the Location term assumes square pixels.
New SalConfig (enabled/params/leadtimes, detection thresholds, raster extent+spacing) under experiment.sal, mirroring ScoreMapsConfig, plus a validate_sal_leadtimes model validator. Regenerate config.schema.json.
verification_sal.py computes per-init SAL over all init times for a given (participant, param, lead time), remapping forecast and truth onto a common raster, and writes a CSV (one row per init: S,A,L + domain means, with a commented metadata header). New verification_sal / verification_sal_baseline rules produce these CSVs for every run and baseline; experiment_all requests them when experiment.sal.enabled (compute-only — plotting lives in the publication-figures workflow). Add an example sal block to varda-single-1.0.yaml (disabled; requires gridded truth).
Cover the raster builder, nearest-neighbour remap, and the SAL wrapper (identical-field zero, dry-window NaN, amplitude sign, location sign, unit-invariance), plus SalConfig parsing/validation and the leadtime validator. 12 tests.
config.schema.json was committed in 57a6b26 without the regenerated SAL entries, so config.py (with SalConfig) and the checked-in schema were out of sync and the pydantic-schema pre-commit hook failed in CI. Regenerate from config.py to add the SalConfig definition and the experiment.sal field.
Add a SAL block for all example configs, and adjust the one for varda-single-1.0.yaml such that it works with INCA.
The SAL rules listed only sal.py + data_input as code inputs, so edits to verification/spatial.py (the remap kernel, imported via verification.sal) or verification/__init__.py would not retrigger SAL, risking stale results. Add both as inputs. Also drop the baseline rule cpus_per_task from 24 (copied from scoremaps) to 2 for the single-threaded KDTree+pysteps job.
Use type(item).model_fields instead of the deprecated instance attribute in the SAL and scoremaps leadtime validators (Pydantic 2.11). Extend the grid_extent validator to reject non-finite values and implausible lat/lon ranges, not just wrong length/ordering.
SAL needs a resolved field; remapping sparse station truth (jretrieve, ~150 points) onto the ~1 km raster yields meaningless scores. Add a point-density guard (point_density_per_km2 + MIN_TRUTH_POINT_DENSITY) that fails fast on the first init. Also guard compute_sal against all-NaN input, drop the unused SEASONS constant, and add unit tests for the density guard and grid_extent.
SAL is defined for precipitation only, but sal.params accepted any parameter and the driver would silently compute meaningless scores on e.g. T_2M. Add a lenient guard requiring params to start with TOT_PREC (period-accumulated like TOT_PREC6 and bare cumulative TOT_PREC both allowed): a SalConfig field validator that fails fast at config load, plus a matching check in verification_sal.py for direct invocations. Add unit tests. If SAL should be deployed later on for different variables as well (which is conceivable but unlikely), this guard would need to be relaxed again.
Extract the shared steps-vs-requested-leadtimes check from the scoremaps and SAL validators into a single _reject_unproducible_leadtimes() helper, parameterized by the config-block label used in the error message. Both validators become thin enabled-guards, so future features validating a leadtimes field reuse one implementation instead of copy-pasting.
They are redundant information, the init hour is given in the first column.
The leadtime-validation dedup (ab3c497) hand-wrapped the _reject_unproducible_leadtimes signature across three lines; ruff collapses it to one. Without this the ruff-format CI hook fails.
…nto paper/sal-figures # Conflicts: # src/evalml/config.py # workflow/tools/config.schema.json
Introduce PublicationSalScatterConfig and PublicationCaseSnapshotsConfig under PublicationConfig, and extend ConfigModel.validate_publication with coherence checks: gridded (zarr) truth, cases within `dates`, producible lead times, and matching experiment.sal params/lead times. Regenerate the config JSON schema.
Add run_sal/baseline_sal path templates and a per-participant sal_template to build_manifest, carry experiment.sal settings into the manifest, and bump the manifest schema version to 2. Add Manifest.sal_path() and validate_request branches for the sal_scatter and case_snapshots figures.
Two `python -m evalml.publication` subcommands that resolve data from the manifest (SAL CSVs for the scatter; candidate GRIB, truth and SAL annotation for the case snapshots) and invoke the figure scripts, mirroring the existing scoremaps/meteogram commands. The figure scripts themselves land in a follow-up commit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
This PR integrates the two SAL-based precipitation figures for the Varda paper as first-class figures in the publication-figures workflow: an S–A (Structure–Amplitude) scatter across the candidate and the ICON baselines, and a case-snapshot map panel (candidate forecast vs KENDA-CH1 analysis) for the selected cases. Both are driven exactly like the existing
figures/scoremaps/meteogramfigures — resolved from the manifest, rendered viapython -m evalml.publication <figure>, and wired intopublication_allbehind per-figureenabledswitches.The SAL scores themselves come from the SAL verification core (compute-only) in the SAL-core PR (
feat/MRB-921); this PR adds only the plotting and the config/manifest/CLI plumbing to consume it.Status — draft / work in progress
Plumbing landed so far:
publication:config —PublicationSalScatterConfigandPublicationCaseSnapshotsConfig, plus coherence checks invalidate_publication(gridded truth, cases withindates, producible lead times, matchingexperiment.salparams/lead times); schema regenerated.run_sal/baseline_salpath templates and a per-participantsal_template;experiment.salsettings carried into the manifest; schema version bumped to 2.Manifest.sal_path()andvalidate_requestbranches for both figures.sal-scatterandcase-snapshotssubcommands.Still to come (tracked in
PLAN_PR2_sal_figures.md):workflow/scripts/publication_sal_scatter.py(incl. the SAL-CSV reshaping adapter) andpublication_case_snapshots.py.publication_sal_scatter/publication_case_snapshotsand their wiring intopublication_all.experiment.sal+publication:blocks inconfig/varda-single_paper_analysis.yaml.--dry-run, standalone smoke test).Relationship to the SAL-core PR and the base branch
This branch targets
paper-figuresand is stacked on it. Because the SAL core is not yet onmain, this branch mergesfeat/MRB-921in so the figures have data to consume. For review that means the diff currently includes the SAL-core changes — please review those in their own PR and focus here on the figure-specific commits. Once the SAL core lands onmainandmainflows intopaper-figures, this branch will be rebased so its diff shows only the figure work (a one-timegit rebase --ontoif the core is squash-merged).Provenance note
Both figures use the config's candidate
Varda-Single(the 1-hourly temporal downscaler — the true Varda output). Scores and the exact case selection may shift slightly once the final model run is available; the selected cases are expected to remain valid and will be re-confirmed then.