Database builder - #217
Conversation
The previous format (separate parameters for detect and stitch) is supported as a fallback.
sjavis
left a comment
sjavis
left a comment
There was a problem hiding this comment.
This looks good and the tests seem pretty comprehensive. I have just a few suggested changes mostly for compatibility with outputs from different tracking algorithms.
| air_pressure_at_sea_level real, | ||
| surface_altitude real, | ||
| wind_speed real, | ||
| atmosphere_relative_vorticity real |
There was a problem hiding this comment.
Ideally the database would only include the columns for the variables that are actually included in the output. Otherwise it could lead to a large number of columns as variables get added down the line. But this seems like it would not be very straightforward so probably not worth worrying about right now.
| "grid_i": netcdf_data["grid_i"][traj_idx], | ||
| "grid_j": netcdf_data["grid_j"][traj_idx], |
There was a problem hiding this comment.
These variables should be optional. I believe only tempest extremes outputs them.
There was a problem hiding this comment.
I'll drop them completely as the dashboard map rendering relies solely on lat/lon. It's best to start with a minimal schema and then add to it on user demand.
| "attributes": {k: ds.getncattr(k) for k in ds.ncattrs()}, | ||
| "latitude": ds.variables["latitude"][:], | ||
| "longitude": ds.variables["longitude"][:], | ||
| "grid_i": ds.variables["grid_i"][:], | ||
| "grid_j": ds.variables["grid_j"][:], | ||
| } | ||
|
|
||
| time_var = ds.variables["time"] | ||
| data["time_units"] = time_var.getncattr("units") | ||
| data["time_calendar"] = time_var.getncattr("calendar") | ||
| data["time"] = ds.variables["time"][:] | ||
|
|
||
| # Read start/end flags if present (per-trajectory booleans) | ||
| if "start_flag" in ds.variables: | ||
| data["start_flag"] = ds.variables["start_flag"][:] | ||
| if "end_flag" in ds.variables: | ||
| data["end_flag"] = ds.variables["end_flag"][:] | ||
|
|
||
| # Observations (optional) | ||
| for prop in [ | ||
| "air_pressure_at_sea_level", | ||
| "surface_altitude", | ||
| "wind_speed", | ||
| "atmosphere_relative_vorticity", | ||
| ]: | ||
| data[prop] = v[:] if (v := ds.variables.get(prop)) is not None else None |
There was a problem hiding this comment.
I suggest that we change this so it selects variables by matching standard_name, then falls back to using the netcdf variable name, eg. replace `ds.variables.get("wind_speed") with a function like:
def get_var(ds, variable_name: str):
for var in ds.variables.values():
if getattr(var, "standard_name", None) == variable_name:
return var
return ds.variables.get(variable_name)
get_var(ds, "wind_speed")Alternatively it could use cf-python to do the reading instead of netcdf4.
There was a problem hiding this comment.
It's a good idea but I've never seen a file where the standard_name and the actual name differed. Do you have any examples?
| # Support layout from previous version | ||
| tctrack_parameters = json.dumps( | ||
| { | ||
| "detect": json.loads(attributes.get("detect_parameters", "{}")), | ||
| "stitch": json.loads(attributes.get("stitch_parameters", "{}")), | ||
| } | ||
| ) |
There was a problem hiding this comment.
| # Support layout from previous version | |
| tctrack_parameters = json.dumps( | |
| { | |
| "detect": json.loads(attributes.get("detect_parameters", "{}")), | |
| "stitch": json.loads(attributes.get("stitch_parameters", "{}")), | |
| } | |
| ) | |
| # Support layout from previous version | |
| tctrack_parameters = json.dumps( | |
| { | |
| key[:-len("_parameters")]: json.loads(value) | |
| for key, value in attributes.items() | |
| if key.endswith("_parameters") | |
| } | |
| ) |
Different tracking algorithms had different names for the *_parameters attributes
2 participants
A CLI tool to import TCTrack NetCDF files into SQLite databases.
The resulting database forms the foundation for a dashboard. However, it is also valuable as a organisational tool for TCTrack output files. The track GeoJSON conversion is useful for visualising in general GIS tools.
Variable aliases to be supported in future.
Tool is pip-installed as
build-db.See README for full details.