Skip to content

Fix silently-ignored grouping, lost metadata on grouped variables, and four smaller defects - #2

Merged
rafa-guedes merged 3 commits into
mainfrom
fix/stat-group-and-metadata-defects
Aug 12, 2026
Merged

Fix silently-ignored grouping, lost metadata on grouped variables, and four smaller defects#2
rafa-guedes merged 3 commits into
mainfrom
fix/stat-group-and-metadata-defects

Conversation

@rafa-guedes

Copy link
Copy Markdown
Contributor

Motivation

Writing a client-facing description of a delivered gridstats product meant reading every stat against its source. Seven defects surfaced, all of the same shape: the library produces a plausible-looking answer instead of an error. Each is fixed here, with a regression test that fails without the fix.

Three commits, grouped by concern. Each passes the suite on its own.


1. group silently ignored — caf862a

CallConfig.group is forwarded to every stat, and pcount, rpv, hmo and distribution3_timestep absorbed it into **kwargs and ignored it. Meanwhile _apply still appended _{group} to the output name. A config asking for group: month on pcount got a variable called hs_pcount_month holding the ungrouped whole-record value, with no month dimension. Wrong data, convincing name, no error — the same defect fixed for range_probability in #1, in four more places.

Group support is now detected by inspecting the function signature rather than a hand-maintained flag: a stat supports grouping if it declares an explicit group parameter rather than absorbing it variadically. That cannot drift out of sync with the code. Pipeline._apply raises before any data is read if grouping is requested from a stat that cannot honour it, naming the stat and listing those that can.

pcount gains real grouping, since it is the one of the four where the meaning is unambiguous — the denominator is the group's own timestep count, not the whole record. rpv, hmo and distribution3_timestep are left unsupported deliberately (a per-calendar-month extreme-value fit on a twelfth of the samples is not something to offer by accident); all three also documented a group argument they ignored, in docstrings and in docs/, now corrected.

harmonics and statdir were also swept in as unsupported by the signature rule. statdir gains support in commit 3; harmonics is genuinely not groupable.

Also fixes rpv's nanosecond assumptionfloat(np.diff(...)) / 3.6e12 raises TypeError under NumPy 2 for any other datetime64 resolution and would silently mis-scale if it did not. Same fix already applied to exceedance.

2. Metadata and naming — 66836a5

Grouped variables lost every attribute. set_variable_attributes treats the last _ token as the statistic, so hs_mean_month resolved as stat month, the lookup failed, and the variable shipped with {} — no units, no names. The existing code meant to handle this was unreachable, tested a substring rather than the trailing token, and would have emitted "seasonly". Now a recognised trailing group token is stripped before resolution, with an explicit adjective mapping (monthly / seasonal / annual), composing correctly with _direc and with the stat_units override.

count shipped as metres. hs_count inherited its parent's units; count and dist are now declared dimensionless in stat_units.

range_probability default labels could contain a doths_1.5_to_max. The p-substitution applied to exceedance now lives in a shared ops/_naming.py used by both, so the two cannot drift.

3. statdir and output.chunks761af67

statdir raised MergeError with more than one func, because each delegated stat returned its result under the base variable name. Outputs are now suffixed with the producing function, giving hs_mean_direc / hs_max_direc — byte-identical to what equivalent nsector calls produce, so the same product has one name by either route and still resolves in attributes.yml.

A second, worse interaction turned up while pinning that: nsector is a CallConfig field, so on a statdir call it was consumed by the pipeline's directional wrapper and never reached the stat, which then re-binned each outer sector with its own default of 4 — the documented eight-sector example died with conflicting dimension sizes. Sector settings are now forwarded to self-sectorising stats rather than wrapped around them. statdir also gains group, validated across all delegated funcs.

statdir is now functionally redundant with nsector, its only remaining justification being one pass over the data for several statistics. Kept, with the redundancy documented rather than left implicit.

output.chunks was silently dropped. pipeline.py read it via __dict__.get, but OutputConfig never declared the field, so pydantic discarded it — finalise() has a working chunks parameter that never received anything. The field is now declared, and OutputConfig rejects unknown keys, since silently swallowing output options is what let this hide.


Breaking changes

  • Requesting group on a stat that cannot honour it now raises instead of being ignored.
  • range_probability default labels render a fractional bound as 1p5 rather than 1.5. Configs setting label: explicitly are unaffected.
  • OutputConfig rejects unknown keys.

I checked the blast radius of the last one: 98 embedded gridstats configs across config/hindcast/prax/stats/ validate cleanly, including all seven in the current Cawthron delivery. The only failures are legacy onstats configs that fail at the YAML-tag level on !!python/object/apply:onstats.stats.Stats — a different package.

Grouped variables written before this branch carry no metadata and need rewriting, or at least re-consolidating, to pick it up. Grouped pcount output produced earlier is wrong and should be regenerated.

Testing

308 passing, up from 231; zero failures. Every fix has a test that fails without it — including a pcount denominator test constructed so the whole-record denominator gives a different, wrong answer, and a statdir pipeline test asserting equality with the equivalent nsector calls.

Docs updated alongside: docs/ops/rpv.md, docs/ops/distribution.md, docs/ops/aggregations.md, docs/ops/directional.md, docs/ops/probability.md, docs/configuration/output.md.

…it for `pcount`

`CallConfig.group` is forwarded to every stat by `pipeline.py::_apply`, and the
pipeline appends `_{group}` to the output variable name regardless of what the
stat does with it. `pcount`, `rpv`, `hmo` and `distribution3_timestep` absorbed
it into `**kwargs` and ignored it, so a `group: month` config produced the
ungrouped whole-record value under a name like `hs_pcount_month`, with no
`month` dimension — wrong data wearing a convincing name, and no error.

`_apply` now raises before any data is loaded when `group` is requested from a
stat that cannot honour it, naming the stat and listing the ones that can. The
check runs ahead of the nsector/tiles branch so it fires for every call. Support
is derived from the function signature by the new `registry.supports_group()`:
a stat supports grouping when it declares an explicit `group` parameter, which
`inspect.signature` reads through the `functools.wraps` registration wrapper.
That cannot drift out of sync with the code the way a curated list would.

`pcount` gains real group support rather than being rejected: the percentage is
computed within each group, with the group's own timestep count as the
denominator, so a gap-free variable reads 100 in every group. Ungrouped
behaviour is unchanged.

Grouping is deliberately not implemented for `rpv`, `hmo` and
`distribution3_timestep`, whose `group` parameters and documentation are
removed: a per-calendar-month extreme-value fit uses a twelfth of the samples
and is not statistically defensible, and grouping is not well defined for a
Welch spectrum or for a histogram accumulated over windows that do not align
with time groups.

Also fixes the nanosecond assumption in `ops/rpv.py`, where the timestep was
derived as `float(np.diff(time)[0]) / 3.6e12`. Under NumPy 2 that raises
`TypeError` for any other `datetime64` resolution, and would silently mis-scale
if it did not. It now goes through `pd.Timedelta`, matching the fix already
applied in `ops/exceedance.py`.
… counts

Grouped output variables were delivered with no attributes at all. The pipeline
appends the group to the output name (`hs_mean` -> `hs_mean_month`), and
attribute resolution read the last `_`-separated token as the statistic, so the
lookup failed and the variable got no units, standard_name or long_name. The
group suffix, and the `_direc` suffix appended after it for a directionally
sectorised call, are now stripped before the statistic and parent variable are
resolved. A grouped variable's long_name is prefixed with an adjective from an
explicit mapping (month -> monthly, season -> seasonal, year -> annual),
replacing dead, unreachable code that would have produced "seasonly" and that
matched a group word anywhere in the name rather than as the trailing token.

`count` and `dist` are added to `stat_units` as "1": both are counts of samples
and were inheriting the units of their parent variable.

The threshold-name formatter that writes a decimal point as `p` moves out of
`ops/exceedance.py` into a shared `ops/_naming.py`, and is now applied to the
default labels `range_probability` builds, so a fractional bound gives
`hs_1p5_to_max` rather than `hs_1.5_to_max`. This is a breaking change for
configs relying on default labels; explicit labels are unaffected and
`exceedance` naming is unchanged.
…group; apply output.chunks

`statdir` delegates to other registered stats per directional sector, but each
one returns its result under the base variable name, so `funcs: [mean, max]`
produced two datasets both containing `hs` and the sector merge raised
`MergeError`. Delegated outputs are now suffixed with the function that
produced them (`hs_mean`, `hs_max`), unconditionally, so a name never depends
on the length of `funcs`. The pipeline no longer appends `_statdir` on top of
that, writing `hs_mean_direc` — the same name the equivalent `nsector` call
produces.

`nsector` on a `statdir` call was consumed by the pipeline's sectoriser and
never reached the stat, which then binned each outer sector again with its own
default of 4; the sector settings are now forwarded to stats that sectorise
internally. `group` is now an explicit `statdir` parameter, so the
signature-based guard admits it, and a delegated stat that cannot group is
reported up front.

`output.chunks` was read with `__dict__.get`, but `OutputConfig` declared no
such field and Pydantic had already dropped it, so output chunking was silently
ignored. It is now a declared `dict[str, int]` field read by attribute access,
and `OutputConfig` forbids unknown fields so the next misspelled output option
fails validation instead of being discarded.
@rafa-guedes
rafa-guedes merged commit f7a1981 into main Aug 12, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant