Conversation
…bles The AMWG tables read their time series files directly, so the scale factor, offset and units named in the variable defaults were never applied and the tables reported the model's raw values: precipitation came out as 3.37e-08 m/s where the defaults ask for mm/day. They now load through the ADF's own data layer, which applies them (issue NCAR#423). The same issue notes that nothing corrected the time stamps. CAM stamps a monthly average with one end of the interval it covers, and which end depends on the model version: an older CAM h0 file stamps January with February 1st. Anything that then asks which year a step belongs to puts it in the wrong one, so a five year request produced a four year average, silently. Rather than correct this in the tables, it is corrected where files are opened, so that a script which does not know about the stamping convention does not have to. The bounds variable is taken from the 'bounds' attribute of the time coordinate, which is where CF says it belongs and is authoritative when present; only if it is missing or names something absent are the conventional names tried. That precedence is not academic here: of the two test cases, one records 'time_bnds' and the other 'time_bounds', and both declare it in the attribute. There were five hand-rolled versions of this in the tree, none of which read the attribute and all of which assumed the bounds dimension is called 'nbnd'. One of them, in create_climo_files, looked only for 'time_bnds', so a model writing 'time_bounds' had its climatologies built from raw stamps too. Four are now the one shared function; the two in the TEM scripts are left alone, since TEM has its own data and its own tests. Measured on a five year model against model run, for a case stamped at the end of the interval: precipitation goes from 3.37e-08 m/s to 2.907 mm/day, and the sample size from four years to five. A case already stamped mid-interval is unchanged, and the climatology files for both cases are identical to those the previous code wrote.
Contributor
Author
|
Reopened as #477 from the right account; same branch and same commit. Closing this one to keep the discussion in one place. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #423.
The AMWG tables were reporting raw model values, and the time stamps in the
time series files were never corrected, so a five year request could produce a
four year average. Both are fixed.
The variable defaults were not applied
The tables read their time series files directly with xarray, so the scale
factor, offset and units from the variable defaults never reached them. They
now load through the ADF's own data layer, which applies them.
The time stamps were not corrected
CAM stamps a monthly average with one end of the interval it covers, and which
end depends on the model version: an older CAM h0 file stamps January with
February 1st. Anything that then asks which year a step belongs to puts it in
the wrong one. In the tables that shows up as a lost year, because the annual
mean drops the two years that then look incomplete.
This is fixed where files are opened rather than in the tables, so that a
script which does not know about the stamping convention does not have to.
adf_utils.use_time_bounds_midpointsets the time coordinate to the midpointof the interval each step covers, and it is applied by
adf_utils.load_dataset,AdfData.load_dataset,AdfData.load_timeseries_dataset,create_climo_filesand
AdfInfo.Which variable holds the bounds is taken from the
boundsattribute of thetime coordinate first, which is where CF says it belongs and is authoritative
when it is there. Only if that attribute is missing, or names something the
file does not contain, are the conventional names
time_bndsandtime_boundstried. A file that records no bounds is returned untouched, sincethere is then nothing better to go on than the stamp itself.
That precedence is not academic. Of the two cases used for testing, one records
time_bndsand the othertime_bounds, and both declare it in the attribute.Five hand-rolled versions of this logic were in the tree, none of which read
the attribute and all of which assumed the bounds dimension is called
nbnd.One of them, in
create_climo_files, looked only fortime_bnds, so a modelwriting
time_boundshad its climatologies built from raw stamps as well.Four are now the one shared function. The two in the TEM scripts are left
alone: TEM has its own data and its own tests, and widening this further would
make the change harder to review. Worth a follow-up.
What changes for users
Measured on a five year model against model run, for a case stamped at the end
of the interval:
m/smm d$^{-1}$The
2.907is the old value times the 86400000 the defaults ask for. The extrayear is the stamping fix: the average now covers the five years that were
requested, which also moves the numbers slightly (
TS288.575 to 288.544).A case already stamped mid-interval is unchanged, and the climatology files
written for both cases are identical to those the previous code wrote, so this
does not disturb runs that were already correct. Anyone whose model writes
time_boundsand stamps at the end of the interval will see theirclimatologies change, because those were being built from the wrong stamps.
Testing
Unit tests: 135 pass, up from 124. Eleven new ones cover the choice of bounds
variable, including a file whose
boundsattribute points at one variablewhile a differently named decoy carries the conventional name, both spellings
with no attribute at all, an attribute naming something absent, a bounds
dimension called something other than
nbnd, data already stamped at themidpoint, files recording no bounds at all, and both the noleap and standard
calendars.
End to end on Casper, model against model over five years, with the before and
after tables in the table above. The run finishes with no errors and writes its
plots, and the climatology files were compared against those from the previous
code and are identical.
One thing worth a reviewer's eye
AdfData.load_dacallssqueeze()on the variable, which the tables' previousdirect read did not. For a variable with a single vertical level that would
drop the level dimension, and the table skips variables by looking for
levinthe coordinates, so such a variable would now be included where it used to be
skipped. No variable in the test configuration is like that, and it is
arguably the better behaviour, but it is a change and it is not something the
issue asked for.