Improve CI by clairemerker · Pull Request #226 · MeteoSwiss/evalml

clairemerker · 2026-07-23T12:17:10Z

Capabilities of evalml that need testing

2 models: forecaster and temporal downscaler
4 flavours of baselines: INCA, ICON-CH1/2-CTRL, ICON-CH1/2-EPS median, and ICON-CH1/2-EPS mean
3 truth datasets: jretrieve (stations), operational analysis, and re-analysis
2 modes:
- experiment: dashboard, plots, scorecard, and scoremaps
- showcases: meteogram and animations
various stratification options (e.g. plotting domain, region specification, etc.)

Proposed solution

We disentangle testing of inference in evalml and testing of evalml core functionality (e.g. experiment reports, meteograms, animations). To do so, we generate reference inference data that we can flexilbly reuse for testing. Heavy end-to-end tests still run the whole workflow including inference.

0. unit tests

Unit tests are run using Github Actions on every push to PRs to main.
Trigger locally with pytest

1. shorter integration tests (to be run with every PR) - longtest

Those tests can be triggered on a PR with the comment cscs-ci run longtestor locally using ci/run-integration-coverage.sh longtest tests/integration. They need to be run before merging a PR.
Here we use baseline data or frozen inference data only, as this is readily available and lengthy inference can be avoided. We aim to set up a minimal configs that test real-world cases.

test_meteogram_small.py with Varda, ICON-CH1-CTRL and ICON-CH2-CTRL, 2024-08-01T00:00, SwissMetNet truth, T_2M/SP_10M/TOT_PREC6, GVE/SAE
test_animation_small.py with ICON-CH2-CTRL (restrict lead times to only 2)
test_dashboard_small.py with ICON-CH1-CTRL and INCA
test_scoremaps_small.py with ICON-CH2-CTRL, 2025-04-01T00:00 - 2025-04-01T06:00, KENDA-CH1 truth, T_2M/TOT_PREC6, bias, Switzerland
test_mec_small.py: Varda forecaster, 2025-08-01T12:00 - 2025-08-02T00:00, mec (without ffv2)

2. long integration tests (to be run weekly) - heavytest

Those tests can be triggered on a PR with the comment cscs-ci run heavytestor locally using ci/run-integration-coverage.sh heavytest tests/integration. They are run nighly on this branch currently using the CI cron.

test_configs.py to test inference and model verification
forecasters-ich1.yaml: forecaster evaluation experiment, 2025-03-01T00:00 - 2025-03-02T00:00, baseline ICON-CH2-EPS, KENDA-CH1 truth, multiple params, regions, thresholds, dashboard, currently no scoremaps
aifs-single.yaml:
forecasters-ich1-oper-fixed.yaml:
forecasters-ich1-oper.yaml:
forecasters-ich1_mec_ffv2.yaml:
varda-single-1.0.yaml: forecaster and temporal downscaler experiment, 2025-01-01T00:00 / 2025-04-01T00:00 / 2025-07-01T00:00 / 2025-10-01T00:00, baselines INCA / ICON-CH2-CTRL / ICON-CH1-CTRL, SwissMetNet truth, scorecards, multiple params, regions, thresholds

Todo and missing tests

testing mean and median in longtest
consolidate CI documentation in main readme instead of the one in the tests folder

lclanzi · 2026-07-23T12:28:25Z

cscs-ci run

jonasbhend · 2026-07-23T13:59:28Z

cscs-ci run

Co-authored-by: Mary McGlohon <Mary.McGlohon@meteoswiss.ch>

Adds an integration test that runs the full evalml experiment pipeline and asserts that key verification metrics match reference values from a known-good run. --------- Co-authored-by: Luca Lanzilao <llanzila@balfrin-ln004.cscs.ch> Co-authored-by: Luca Lanzilao <llanzila@balfrin-ln003.cscs.ch>

This PR implements minor changes to the existing meteogram integration test. ### Summary of changes * rename test and config to meteogram_small * change date to 2024-08-01 (official start of ICON archive) * additional parameters to cover all use cases (derived, aggregated and instantaneous)

lclanzi · 2026-07-24T14:57:27Z

cscs-ci run

lclanzi · 2026-07-27T07:27:23Z

cscs-ci run

lclanzi · 2026-07-27T09:15:41Z

cscs-ci run

lclanzi · 2026-07-27T09:19:17Z

cscs-ci run

Under the service account, uv needs to be installed for every CI run since there is no global installation. --------- Co-authored-by: Mary McGlohon <Mary.McGlohon@meteoswiss.ch> Co-authored-by: clairemerker <34312518+clairemerker@users.noreply.github.com>

Two different pipelines: - longtest: intergration tests - heavytest: heavy integration tests (full end to end) The idea is to trigger heavytest weekly or nighly using the CI cron, and longtest in PRs using the comment call.

Fix truth in config test move configs to test folder --------- Co-authored-by: Luca Lanzilao <llanzila@balfrin-ln004.cscs.ch>

lclanzi · 2026-07-27T13:10:53Z

cscs-ci run heavytest

… testing-hackathon

lclanzi · 2026-07-27T13:29:40Z

cscs-ci run heavytest

lclanzi · 2026-07-27T14:36:27Z

cscs-ci run heavytest

lclanzi · 2026-07-28T07:28:27Z

cscs-ci run heavytest

Adds test coverage diagnostics - to the unit tests - to the integration tests with subprocess coverage

clairemerker · 2026-07-28T10:07:11Z

cscs-ci run longtest

expose logs from evalml from pipeline to balfrin --------- Co-authored-by: Luca Lanzilao <llanzila@balfrin-ln004.cscs.ch>

lclanzi · 2026-07-28T16:25:09Z

cscs-ci run longtest

lclanzi · 2026-07-28T16:25:17Z

cscs-ci run heavytest

AI assisted implementation! Adds an opt-in mechanism to replay pre-computed inference output instead of running inference, so the integration tests can run without needing inference. Truth and baselines still come from the DWH and /store_new: only inference is frozen. Adds a new top-level config key fixture_root (absent by default → production unchanged): - unset: inference_execute runs anemoi-inference exactly as previously - set: inference_execute symlinks frozen GRIB from <fixture_root>/data/runs/<run_id>/<init_time>/grib into the run workdir and marks the step done

Adds testing config for scoremaps. --------- Co-authored-by: clairemerker <34312518+clairemerker@users.noreply.github.com>

clairemerker · 2026-07-29T11:44:04Z

cscs-ci run longtest

lclanzi · 2026-07-29T14:10:43Z

cscs-ci run heavytest

lclanzi · 2026-07-29T14:10:51Z

cscs-ci run longtest

- New tests/integration/test_dashboard_small.py + dashboard_small.yaml, exercising the dashboard pipeline against a frozen inference fixture - Move fixture_root for dashboard/meteogram small tests to a shared team location under /store_new - small cleanup to tools/summarize_logs.py to make benign error-checking less bulky

clairemerker · 2026-07-29T11:44:52Z

@@ -0,0 +1,133 @@
+#!/usr/bin/env python3


Should we move this to the ci folder?

It depends, if you only want to use it for ci probably yes. If it might be useful for users and developers, here might be good.
The question is if we want to reuse it for all snakemake projects, then it be cool if was in its own repo or kind of a utility repo that then all snakemake repos can use.

clairemerker · 2026-07-29T11:47:51Z

+    - |
+      banner "Install uv"
+      export UV_CACHE_DIR="$SCRATCH/.cache/uv"  # wheel cache, persistent across pipelines
+      # Set eckit ECKIT_GEO_CACHE_PATH for grid downloads


This is not really related to uv, rename the script or put somewhere else?

clairemerker · 2026-07-29T12:46:00Z

@@ -1,4 +1,4 @@
-lead_time: 120h
+lead_time: 12h


Revert, has no effect on testing anyways I think.

Suggested change

lead_time: 12h

lead_time: 120h

clairemerker · 2026-07-29T12:55:38Z

+
+# # yaml-language-server: $schema=../../../workflow/tools/config.schema.json
+# description: |
+#   Evaluate skill of ICON-CH1 single.


Suggested change

# Evaluate skill of ICON-CH1 single.

# Evaluate skill of Varda-single hierarchical 2 level mesh.

clairemerker · 2026-07-29T12:56:46Z

+
+# # yaml-language-server: $schema=../../../workflow/tools/config.schema.json
+# description: |
+#   Evaluate skill of ICON-CH1 single.


Suggested change

# Evaluate skill of ICON-CH1 single.

# Evaluate skill of Varda-single forecaster.

clairemerker · 2026-07-29T13:02:26Z

-# yaml-language-server: $schema=../workflow/tools/config.schema.json
+# yaml-language-server: $schema=../../../workflow/tools/config.schema.json
 description: |
  Evaluate skill of ICON-CH1 single.


Suggested change

Evaluate skill of ICON-CH1 single.

Evaluate skill of Varda-single forecaster.

huppd · 2026-07-30T09:11:37Z

+      export PATH="$PWD/.local/bin:$PATH"
+      # uv: bare metal assumes a user-managed env so evalml ships no installer
+      export UV_INSTALL_DIR="$PWD/.local/bin"
+      curl -LsSf https://astral.sh/uv/install.sh | env INSTALLER_NO_MODIFY_PATH=1 sh


I would suggest to pin the uv version:

Suggested change

curl -LsSf https://astral.sh/uv/install.sh | env INSTALLER_NO_MODIFY_PATH=1 sh

UV_VERSION='0.11.32'

curl -LsSf "https://astral.sh/uv/${UV_VERSION}/install.sh" | env INSTALLER_NO_MODIFY_PATH=1 sh

huppd

I haven't looked into it in greate detail. But I like the split of the ci yamls, and the nice READMEs.

huppd · 2026-07-30T11:25:16Z

I have a slight feeling that this and the .cfg could be also in the tests folder. Maybe something to think about. I am not sure which is the better place.

huppd · 2026-07-30T12:54:29Z

@@ -0,0 +1,133 @@
+#!/usr/bin/env python3


It depends, if you only want to use it for ci probably yes. If it might be useful for users and developers, here might be good.
The question is if we want to reuse it for all snakemake projects, then it be cool if was in its own repo or kind of a utility repo that then all snakemake repos can use.

huppd · 2026-07-30T12:55:35Z

+DEST = ROOT / "tests" / "log"
+DEST.mkdir(exist_ok=True)
+
+ERROR_KEYWORDS = {"error", "exception", "traceback", "failed", "oom", "killed"}


A friend told me we might want to add

Suggested change

ERROR_KEYWORDS = {"error", "exception", "traceback", "failed", "oom", "killed"}

ERROR_KEYWORDS = {"error", "exception", "traceback", "failed", "oom", "killed", "cancel"}

When heavytests fail, produce some stats. See below for example: varda-single-1.0.yaml: 2898/3326 pass, 428/3326 fail (13%) | tol=1e-02 | diff range [0.00e+00 – 3.07e+00], median 4.69e-07 | failed vars: PMSL(72), SP_10M(140), TD_2M(48), TOT_PREC1(8), TOT_PREC6(14), T_2M(146) Co-authored-by: Luca Lanzilao <llanzila@balfrin-ln004.cscs.ch>

lclanzi · 2026-07-30T14:28:31Z

cscs-ci run heavytest

- test running evalml --mec and asserting verSYNOP_*.nc files are produced (inference frozen) - resources/mec/namelist.jinja2: switch prefix_in to ekf - workflow/rules/verif_obs.smk: guard FFV2 rules under if config[ffv2] - workflow/Snakefile: guard ffv2_all under mec+ffv2 both non-None --------- Co-authored-by: clairemerker <34312518+clairemerker@users.noreply.github.com>

minor change

9ede394

enable longtests @cscs

87b3b3b

clairemerker and others added 6 commits July 24, 2026 09:12

Merge branch 'main' into testing-hackathon

b0a21e0

increase time limit for integration tests

a5893b3

fix lint error (#230)

4bfd1bc

Co-authored-by: Mary McGlohon <Mary.McGlohon@meteoswiss.ch>

Merge branch 'main' into testing-hackathon

5f26b34

clairemerker added 3 commits July 24, 2026 17:05

linting

9ca7883

exclude longtest and heavytest from default pytest run

b6b0c75

fix syntax

e29d474

marymcglo and others added 5 commits July 27, 2026 13:42

edit config test and move configs to test folder

221160f

split CI into two pipelines (#237)

f1092ef

Two different pipelines: - longtest: intergration tests - heavytest: heavy integration tests (full end to end) The idea is to trigger heavytest weekly or nighly using the CI cron, and longtest in PRs using the comment call.

reduce verbose

0f444a8

edit config test and move configs to test folder (#238)

c1131c3

Fix truth in config test move configs to test folder --------- Co-authored-by: Luca Lanzilao <llanzila@balfrin-ln004.cscs.ch>

Merge branch 'testing-hackathon' of github.com:MeteoSwiss/evalml into…

98b625b

… testing-hackathon

Luca Lanzilao added 4 commits July 27, 2026 16:17

allow more characters in stderr

dd61117

Merge branch 'main' into testing-hackathon

80ad071

fix linting

4a37e40

fix unit test

cf9b967

minor edit to config

818cc80

Add pytest-cov coverage reporting to test suite (#228)

b17aae8

Adds test coverage diagnostics - to the unit tests - to the integration tests with subprocess coverage

lclanzi and others added 4 commits July 28, 2026 15:55

add logging in pipeline (#241)

7ea3bd5

expose logs from evalml from pipeline to balfrin --------- Co-authored-by: Luca Lanzilao <llanzila@balfrin-ln004.cscs.ch>

add artifacts paths to longtest pipeline

83053bf

add baseline to config and fix linting

fa0ea84

writable location for ECKIT_GEO_CACHE_PATH (#242)

92e336d

clairemerker and others added 5 commits July 29, 2026 08:55

Merge branch 'main' into testing-hackathon

3a84163

Add small integration test for scoremaps (#227)

3353f10

Adds testing config for scoremaps. --------- Co-authored-by: clairemerker <34312518+clairemerker@users.noreply.github.com>

implement fixture versioning

3b184e7

temporary fix for caching race condition triggered in CI tests

7b2ba38

edit ground truth in config and decrease tolerance

e75c5e8

clairemerker and others added 4 commits July 29, 2026 16:15

add allowlist to avoid failing log icon for eckit warning

c3e4802

add newline

4b9bf78

other lint

4963cc1

clairemerker commented Jul 30, 2026

View reviewed changes

huppd reviewed Jul 30, 2026

View reviewed changes

	# Evaluate skill of ICON-CH1 single.
	# Evaluate skill of Varda-single hierarchical 2 level mesh.

	# Evaluate skill of ICON-CH1 single.
	# Evaluate skill of Varda-single forecaster.

	Evaluate skill of ICON-CH1 single.
	Evaluate skill of Varda-single forecaster.

	curl -LsSf https://astral.sh/uv/install.sh \| env INSTALLER_NO_MODIFY_PATH=1 sh
	UV_VERSION='0.11.32'
	curl -LsSf "https://astral.sh/uv/${UV_VERSION}/install.sh" \| env INSTALLER_NO_MODIFY_PATH=1 sh

	ERROR_KEYWORDS = {"error", "exception", "traceback", "failed", "oom", "killed"}
	ERROR_KEYWORDS = {"error", "exception", "traceback", "failed", "oom", "killed", "cancel"}

Uh oh!

Conversation

clairemerker commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Capabilities of evalml that need testing

Proposed solution

0. unit tests

1. shorter integration tests (to be run with every PR) - longtest

2. long integration tests (to be run weekly) - heavytest

Todo and missing tests

Uh oh!

lclanzi commented Jul 23, 2026

Uh oh!

jonasbhend commented Jul 23, 2026

Uh oh!

lclanzi commented Jul 24, 2026

Uh oh!

lclanzi commented Jul 27, 2026

Uh oh!

lclanzi commented Jul 27, 2026

Uh oh!

lclanzi commented Jul 27, 2026

Uh oh!

lclanzi commented Jul 27, 2026

Uh oh!

lclanzi commented Jul 27, 2026

Uh oh!

lclanzi commented Jul 27, 2026

Uh oh!

lclanzi commented Jul 28, 2026

Uh oh!

clairemerker commented Jul 28, 2026

Uh oh!

lclanzi commented Jul 28, 2026

Uh oh!

lclanzi commented Jul 28, 2026

Uh oh!

clairemerker commented Jul 29, 2026

Uh oh!

lclanzi commented Jul 29, 2026

Uh oh!

lclanzi commented Jul 29, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

huppd left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lclanzi commented Jul 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

clairemerker commented Jul 23, 2026 •

edited

Loading