Add ensemble functionality

[KristianHMoller]

The underlying probabilistic advection method has the option of running an ensemble with noise. However, until now, this option has not been implemented into sunflow.

This PR allows for running as an ensemble by specifying the CLI argument --ensemble_members to be different from one. Per default, the output will be selected ensemble statistics, but specifying --full_ensemble when calling sunflow will instead provide the full ensemble as output. The output_mode global attribute in the NetCDF file will specify the type of output: full ensemble, ensemble_statistic or deterministic.

If only a subset of the ensemble statistics are desired, this can be specified by exporting the environment variable ENSEMBLE_STATISTICS with the desired variables comma separated.

The default values of the noise parameters alpha and beta are 0 for deterministic mode and using other values will result in a warning.
For ensemble mode, currently, the default values are taken from Carpentieri et al.: https://www.sciencedirect.com/science/article/pii/S030626192301139X

[KristianHMoller]

Scientific question 1: Which values to use for alpha and beta?
Currently using optimal values from Carpentieri et al., but these may be country-specific and are presumably derived with the potential error in the implementation (dmidk/SolarSTEPS#2).
Also: @irenelivia , we used slightly different values in our previous code. Do you recall why?

[KristianHMoller]

Implementation question 1: Do we want warning or failure if running deterministically (only one ensemble member) with noise?

[KristianHMoller]

Implementation question 2: Do we want the number of ensemble members and noise parameters to be CLI arguments rather than environemnt variables? I.e. parameters called when running sunflow: sunflow --ens_members 10?

Current decision: CLI argument

[KristianHMoller]

Ideas for ensemble spread variables: Quantiles, 10/90 percentile
Implemented as default options: 10, 25, 75 and 90th percentiles

[KristianHMoller]

Mean vs. median - which is more useful?
Current decision: Default uses both

[KristianHMoller]

@irenelivia If you want, you can try this out to see if anything obvious stands out as ideas for improvements or things that should be changed.

Ideas for simple commands to test:
source .venv/bin/activate

sunflow --ensemble_members 10

sunflow --ensemble_members 10 --full_ensemble

export ENSEMBLE_STATISTICS=mean,median
sunflow --ensemble_members 10

[irenelivia]

Awesome work @KristianHMoller .

Three improvements suggested to strengthen this PR:

(1) Little validation bug in main.py:378 - the clearsky completeness check should include the t=0 timestamp (all_clearsky_time_steps instead of previous_day_time_steps), preventing crashes when t=0 data is missing.

(2) Clarify docstring in forecast.py:89-90 to document that return shape varies by ens_members (single-member: 3D, multi-member: 4D), eliminating confusion for future users.

(3) Add dimension check in data_io.py to validate that ensemble array dimensions match the configured ens_members count, catching programmer errors early if validation logic changes.

I tested:

1. sunflow --ensemble_members 10 --time 2026-07-17T09:15Z --full_ensemble

Works seamlessly with the ensemble dimension. Nice! Very interesting to skim through the different noise images.
(Note: this still has the "arctan artifacts")

2. sunflow --ensemble_members 10 --time 2026-07-17T09:15Z

Also works fine, just a small comment to change the output names of the statistical variables, so the statistic comes first in the name (easier to see on ncview for oldies like me 😄)
(Note: this still has the "arctan artifacts")

3. export ENSEMBLE_STATISTICS=mean,median
   sunflow --ensemble_members 10 --time 2026-07-17T09:15Z

Also works fine, outputting only mean and median statistics.
(Note: this still has the "arctan artifacts")

4. sunflow --ensemble_members 0 --time 2026-07-17T09:15Z

Works fine, outputs 1 ensemble member, no arctan artifacts

6. sunflow --ensemble_members 1 --time 2026-07-17T09:15Z

Works fine, outputs 1 ensemble member, no arctan artifacts

[KristianHMoller]

Mean and percentiles do not seem to cover full domain, while median does. Investigate why!

EDIT: Bug solved with: dac8168

[KristianHMoller]

Significant speed-up should be possible by doing cropping to domain_nowcast directly following the forecast calculation.

EDIT: Implemented with: dac8168

[KristianHMoller]

Thanks a lot for thorough review and testing @irenelivia
I implemented a check to make sure that the number of ensemble members given is greater than 0:
7422fc3

This should be ready for re-review :)

[irenelivia]

I re-tested the updated pr17 using ENSEMBLE_STATISTICS="median,p10,p90" uv run sunflow --run_mode files --ensemble_members 5 --time 2026-08-06T09:30Z
. Works well, and very nice now with the statistics showing up first in the name! See my old-school ncview:

minor comments now from a quick re-review with Claude. Can be merged afterwards!