Single in_data file for DetectNode and NodeFileEditor

[NihatCubukcu]

Hi
Kind of related to #40 and #76, according to the paper in here, we should be able to use the same input file (even if it has extra fields) as
DetectNodes --in_data_list files.txt --out_file_list OutputFiles.txt --timefilter "6hr" --mergedist 6.0 --searchbymin mean_sea_level_pressure --closedcontourcmd "mean_sea_level_pressure,200.0,5.5,
0;_DIFF(geopotential(300hPa),geopotential(500hPa)),-58.8,6.5,1.0" --outputcmd "mean_sea_level_pressure,min,0;_DIV(geopotential_at_surface,9.81),max,0;_VECMAG(VAR_10U,VAR_10V),max,2" --latname lat --lonname lon --regional
StitchNodes --in_list OutputFiles.txt --out Tracks.csv --out_file_format "csv" --in_fmt "lon,lat,slp,zs,wind" --range 8.0 --mintime "12h" --maxgap "12h" --threshold "wind,>=,5.0,2;lat,<=,50.0,2;la
t,>=,-50.0,2;zs,<=,150.0,4"
NodeFileEditor --in_data_list files.txt --in_nodefile OutputFiles.txt --in_nodefile_type "SN" --in_fmt "lon,lat,slp,zs,wind" --out_nodefile rmax.txt --out_fmt "lon,lat,rsize,rprof" --timefilter "6hr" --calculate "rprof=radial_wind_profile (VAR_10U,VAR_10V,159,0.125);rsize=lastwhere(rprof,>,8)" --latname lat --lonname lon --regional

Here files.txt is used both in DetectNodes and NodeFileEditor and its content is :

mean_sea_level_pressure.nc;VAR_10U.nc;VAR_10V.ncgeopotential_at_surface.nc;geopotential.nc;total_precipitation.nc

if this is used, DetectNodes and StitchNodes run just fine. But NodeFileEditor gives "Variable "time" has invalid type in file "mean_sea_level_pressure.nc".

Upon going through NodeFileEditor.cpp code, I decided to to create a separate input file with time dimension as "int64" instead of "datetime64". I rerun NodeFileEditor with the new input file but this time I get
missing "calendar" attribute

I feel like there is an argument missing in my call that prevents me from using the same input file. Any help would be greatly appreciated. Thanks

[zarzycki]

I am not 100% sure what you mean by "same input file."

The former error (as you noted) is almost certainly because datetime64 isn't recognized as a time type.

The latter error is because the "calendar" attribute is not attached to the time coordinate of the NetCDF file. Since many climate models run with noleap calendars (365 or 366 days per year, every year), this is required to parse the time variable correctly.

See the CF data standards here: https://cfconventions.org/Data/cf-conventions/cf-conventions-1.11/cf-conventions.html#time-coordinate

If you have NCO, you can append a calendar attribute like so:

ncatted -a calendar,time,o,c,'standard' mean_sea_level_pressure.nc

This can also be done via xarray, NCL, CDO, etc. Note you will also need a valid "units" array if you do not have one (e.g., "days since 1900-01-01 00:00")

TE could be modified to fall back to a default calendar, but to me that would be risky since a "noleap" vs. "standard" calendar would get out of sync quickly for multi-year datasets. It is better to throw an error and request the user fix. Another option could be to have a command line flag that allows the user to specify the calendar external to the NetCDF file, but that also feels like a bandaid on the CF compliance (or lacktherof) issue.

[NihatCubukcu]

Thanks for the reponse @zarzycki. By "same input file" I mean just like
DetectNodes
--in_data_list ERA5_TC_files.txt

and
NodeFileEditor
--in_data_list ERA5_TC_files.txt

as in the paper. As you see ERA5_TC_files.txt is used both in DetectNodes and NodeFileEditor. The content of ERA5_TC_files.txt can be the same I presume (both list of files and formatting) . In my case it is:
: cat files.txt
mean_sea_level_pressure.nc;VAR_10U.nc;VAR_10V.ncgeopotential_at_surface.nc;geopotential.nc;total_precipitation.nc

This content works fine for DetectNodes but it complaints about time dimension in NodeFileEditor.

here is ncdump of one file in the list:
netcdf aifs_VAR_10U {
dimensions:
lat = 200 ;
lon = 398 ;
time = 41 ;
variables:
float lat(lat) ;
lat:_FillValue = NaNf ;
lat:long_name = "latitude" ;
lat:units = "degrees_north" ;
float lon(lon) ;
lon:_FillValue = NaNf ;
lon:long_name = "longitude" ;
lon:units = "degrees_east" ;
double heightAboveGround ;
heightAboveGround:_FillValue = NaN ;
heightAboveGround:long_name = "height above the surface" ;
heightAboveGround:units = "m" ;
heightAboveGround:positive = "up" ;
heightAboveGround:standard_name = "height" ;
double meanSea ;
meanSea:_FillValue = NaN ;
meanSea:long_name = "original GRIB coordinate for key: level(meanSea)" ;
meanSea:units = "1" ;
double surface ;
surface:_FillValue = NaN ;
surface:long_name = "original GRIB coordinate for key: level(surface)" ;
surface:units = "1" ;
int64 time(time) ;
time:units = "hours since 2024-08-09 00:00:00" ;
time:calendar = "proleptic_gregorian" ;
float VAR_10U(time, lat, lon) ;

[zarzycki]

I would check to ensure all the files have a valid (and matching) calendar_attribute.

You may have a static file in that chain without a time dimension (e.g., surface geopotential). If you don't have a time dimension on a file in DN, it will assume a static field and essentially reshape that across the time dimension of the other files. But I assume AutoCurator would bark in NFC.

[NihatCubukcu]

Thank you for the input.. The xarray NetCDF writer in my python code has the default behavior to convert the time dimension to units "hours since [reference time]" for storage efficiency. However, if I want to keep the time dimension in the original datetime format, I have to adjust the time encoding explicitly when saving the dataset. This was hard to catch since DN and SN works with hours unit but NFE did not work. After adjusting the time, it all work nicely. Thank you very much..

[chuaxr]

@ncubukcu, would you be able to share how you adjusted the time encoding? I think I am facing a similar issue.

[NihatCubukcu]

Hi @chuaxr
I use a time list like this:
hours = range(0,246,6)
time=[fcstTime + pd.Timedelta(hours=h) for h in hours]

where
today = date.today()
fcstTime = datetime(today.year, today.month, today.day, int(hour), 0, 0)

when you are building your data, time dimension should have values from this list (see below). If you share the error you are getting and code snippet, I will be happy to help if I can..

def getData(data_batch,home,fcstTime,lab):
hours = range(0,246,6)
time=[fcstTime + pd.Timedelta(hours=h) for h in hours]
ii=0
dSet=None
for hr in hours:
file=home+lab+'-'+fcstTime.strftime('%Y%m%d-%H')+str(hr)+'.nc'
print(file)
if not os.path.isfile(file):
dfile =getForecast(home,fcstTime,str(hr),input_variables,lab)
dfile.to_netcdf(file)
data = dfile.sel(level=levels)
else:
data = xarray.load_dataset(file)
data=data.assign_coords(time=time[ii]).expand_dims('time')
if dSet is None:
file_in=file
dSet = data
else:
dSet = xarray.concat([dSet, data], dim='time')
ii+=1