Fix silently-ignored grouping, lost metadata on grouped variables, and four smaller defects - #2
Merged
Conversation
…it for `pcount`
`CallConfig.group` is forwarded to every stat by `pipeline.py::_apply`, and the
pipeline appends `_{group}` to the output variable name regardless of what the
stat does with it. `pcount`, `rpv`, `hmo` and `distribution3_timestep` absorbed
it into `**kwargs` and ignored it, so a `group: month` config produced the
ungrouped whole-record value under a name like `hs_pcount_month`, with no
`month` dimension — wrong data wearing a convincing name, and no error.
`_apply` now raises before any data is loaded when `group` is requested from a
stat that cannot honour it, naming the stat and listing the ones that can. The
check runs ahead of the nsector/tiles branch so it fires for every call. Support
is derived from the function signature by the new `registry.supports_group()`:
a stat supports grouping when it declares an explicit `group` parameter, which
`inspect.signature` reads through the `functools.wraps` registration wrapper.
That cannot drift out of sync with the code the way a curated list would.
`pcount` gains real group support rather than being rejected: the percentage is
computed within each group, with the group's own timestep count as the
denominator, so a gap-free variable reads 100 in every group. Ungrouped
behaviour is unchanged.
Grouping is deliberately not implemented for `rpv`, `hmo` and
`distribution3_timestep`, whose `group` parameters and documentation are
removed: a per-calendar-month extreme-value fit uses a twelfth of the samples
and is not statistically defensible, and grouping is not well defined for a
Welch spectrum or for a histogram accumulated over windows that do not align
with time groups.
Also fixes the nanosecond assumption in `ops/rpv.py`, where the timestep was
derived as `float(np.diff(time)[0]) / 3.6e12`. Under NumPy 2 that raises
`TypeError` for any other `datetime64` resolution, and would silently mis-scale
if it did not. It now goes through `pd.Timedelta`, matching the fix already
applied in `ops/exceedance.py`.
… counts Grouped output variables were delivered with no attributes at all. The pipeline appends the group to the output name (`hs_mean` -> `hs_mean_month`), and attribute resolution read the last `_`-separated token as the statistic, so the lookup failed and the variable got no units, standard_name or long_name. The group suffix, and the `_direc` suffix appended after it for a directionally sectorised call, are now stripped before the statistic and parent variable are resolved. A grouped variable's long_name is prefixed with an adjective from an explicit mapping (month -> monthly, season -> seasonal, year -> annual), replacing dead, unreachable code that would have produced "seasonly" and that matched a group word anywhere in the name rather than as the trailing token. `count` and `dist` are added to `stat_units` as "1": both are counts of samples and were inheriting the units of their parent variable. The threshold-name formatter that writes a decimal point as `p` moves out of `ops/exceedance.py` into a shared `ops/_naming.py`, and is now applied to the default labels `range_probability` builds, so a fractional bound gives `hs_1p5_to_max` rather than `hs_1.5_to_max`. This is a breaking change for configs relying on default labels; explicit labels are unaffected and `exceedance` naming is unchanged.
…group; apply output.chunks `statdir` delegates to other registered stats per directional sector, but each one returns its result under the base variable name, so `funcs: [mean, max]` produced two datasets both containing `hs` and the sector merge raised `MergeError`. Delegated outputs are now suffixed with the function that produced them (`hs_mean`, `hs_max`), unconditionally, so a name never depends on the length of `funcs`. The pipeline no longer appends `_statdir` on top of that, writing `hs_mean_direc` — the same name the equivalent `nsector` call produces. `nsector` on a `statdir` call was consumed by the pipeline's sectoriser and never reached the stat, which then binned each outer sector again with its own default of 4; the sector settings are now forwarded to stats that sectorise internally. `group` is now an explicit `statdir` parameter, so the signature-based guard admits it, and a delegated stat that cannot group is reported up front. `output.chunks` was read with `__dict__.get`, but `OutputConfig` declared no such field and Pydantic had already dropped it, so output chunking was silently ignored. It is now a declared `dict[str, int]` field read by attribute access, and `OutputConfig` forbids unknown fields so the next misspelled output option fails validation instead of being discarded.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Writing a client-facing description of a delivered gridstats product meant reading every stat against its source. Seven defects surfaced, all of the same shape: the library produces a plausible-looking answer instead of an error. Each is fixed here, with a regression test that fails without the fix.
Three commits, grouped by concern. Each passes the suite on its own.
1.
groupsilently ignored —caf862aCallConfig.groupis forwarded to every stat, andpcount,rpv,hmoanddistribution3_timestepabsorbed it into**kwargsand ignored it. Meanwhile_applystill appended_{group}to the output name. A config asking forgroup: monthonpcountgot a variable calledhs_pcount_monthholding the ungrouped whole-record value, with nomonthdimension. Wrong data, convincing name, no error — the same defect fixed forrange_probabilityin #1, in four more places.Group support is now detected by inspecting the function signature rather than a hand-maintained flag: a stat supports grouping if it declares an explicit
groupparameter rather than absorbing it variadically. That cannot drift out of sync with the code.Pipeline._applyraises before any data is read if grouping is requested from a stat that cannot honour it, naming the stat and listing those that can.pcountgains real grouping, since it is the one of the four where the meaning is unambiguous — the denominator is the group's own timestep count, not the whole record.rpv,hmoanddistribution3_timestepare left unsupported deliberately (a per-calendar-month extreme-value fit on a twelfth of the samples is not something to offer by accident); all three also documented agroupargument they ignored, in docstrings and indocs/, now corrected.harmonicsandstatdirwere also swept in as unsupported by the signature rule.statdirgains support in commit 3;harmonicsis genuinely not groupable.Also fixes
rpv's nanosecond assumption —float(np.diff(...)) / 3.6e12raisesTypeErrorunder NumPy 2 for any otherdatetime64resolution and would silently mis-scale if it did not. Same fix already applied toexceedance.2. Metadata and naming —
66836a5Grouped variables lost every attribute.
set_variable_attributestreats the last_token as the statistic, sohs_mean_monthresolved as statmonth, the lookup failed, and the variable shipped with{}— no units, no names. The existing code meant to handle this was unreachable, tested a substring rather than the trailing token, and would have emitted "seasonly". Now a recognised trailing group token is stripped before resolution, with an explicit adjective mapping (monthly/seasonal/annual), composing correctly with_direcand with thestat_unitsoverride.countshipped as metres.hs_countinherited its parent's units;countanddistare now declared dimensionless instat_units.range_probabilitydefault labels could contain a dot —hs_1.5_to_max. Thep-substitution applied toexceedancenow lives in a sharedops/_naming.pyused by both, so the two cannot drift.3.
statdirandoutput.chunks—761af67statdirraisedMergeErrorwith more than one func, because each delegated stat returned its result under the base variable name. Outputs are now suffixed with the producing function, givinghs_mean_direc/hs_max_direc— byte-identical to what equivalentnsectorcalls produce, so the same product has one name by either route and still resolves inattributes.yml.A second, worse interaction turned up while pinning that:
nsectoris aCallConfigfield, so on astatdircall it was consumed by the pipeline's directional wrapper and never reached the stat, which then re-binned each outer sector with its own default of 4 — the documented eight-sector example died with conflicting dimension sizes. Sector settings are now forwarded to self-sectorising stats rather than wrapped around them.statdiralso gainsgroup, validated across all delegated funcs.statdiris now functionally redundant withnsector, its only remaining justification being one pass over the data for several statistics. Kept, with the redundancy documented rather than left implicit.output.chunkswas silently dropped.pipeline.pyread it via__dict__.get, butOutputConfignever declared the field, so pydantic discarded it —finalise()has a workingchunksparameter that never received anything. The field is now declared, andOutputConfigrejects unknown keys, since silently swallowing output options is what let this hide.Breaking changes
groupon a stat that cannot honour it now raises instead of being ignored.range_probabilitydefault labels render a fractional bound as1p5rather than1.5. Configs settinglabel:explicitly are unaffected.OutputConfigrejects unknown keys.I checked the blast radius of the last one: 98 embedded gridstats configs across
config/hindcast/prax/stats/validate cleanly, including all seven in the current Cawthron delivery. The only failures are legacyonstatsconfigs that fail at the YAML-tag level on!!python/object/apply:onstats.stats.Stats— a different package.Grouped variables written before this branch carry no metadata and need rewriting, or at least re-consolidating, to pick it up. Grouped
pcountoutput produced earlier is wrong and should be regenerated.Testing
308 passing, up from 231; zero failures. Every fix has a test that fails without it — including a
pcountdenominator test constructed so the whole-record denominator gives a different, wrong answer, and astatdirpipeline test asserting equality with the equivalentnsectorcalls.Docs updated alongside:
docs/ops/rpv.md,docs/ops/distribution.md,docs/ops/aggregations.md,docs/ops/directional.md,docs/ops/probability.md,docs/configuration/output.md.