Summary
"DDH5" is used pervasively across labcore, but the documentation never explicitly defines what it is or specifies its on-disk schema. Today a reader has to reverse-engineer the format from the storage code. We should document it directly — it's foundational knowledge.
The core insight
"DDH5" actually refers to two distinct things that are easy to conflate:
- The DataDict — the in-memory Python object.
- The DDH5 on-disk format — an HDF5 file following a specific schema, which is independent of the DataDict object. You can read and write this layout with plain
h5py without ever constructing a DataDict.
The docs should make this distinction explicit and then specify the on-disk schema precisely. This is what makes interop clear: any tool can read/write DDH5 directly, and the format (not the object) is what plottr, autoplot, and the analysis tooling all rely on.
What to document
Definition of DDH5 (HDF5 file following the DataDict storage schema) and how it relates to — but is not the same as — the DataDict object.
The on-disk schema (exact details should be pulled from src/labcore/data/datadict_storage.py), covering at least:
- top-level group (default
data) holding one HDF5 dataset per field
- dataset attributes:
axes (independent-axis names, for dependents), unit, label
- how independents vs dependents are distinguished (presence/absence of
axes)
- group-level attributes for global/dataset meta, and the
__key__ double-underscore convention for internal meta keys
- creation/close-time attributes
- handling of complex data and array-valued (multi-dimensional) fields
Plus:
- a small annotated example of a real file (e.g. an
h5dump-style listing or a diagram)
- placement in the API docs, linked from the data-storage section of the user guide
Why it matters
The format-vs-object distinction is genuinely illuminating and currently lives only implicitly in the code. Writing it down makes the storage layer much easier to reason about and to build on.
Summary
"DDH5" is used pervasively across labcore, but the documentation never explicitly defines what it is or specifies its on-disk schema. Today a reader has to reverse-engineer the format from the storage code. We should document it directly — it's foundational knowledge.
The core insight
"DDH5" actually refers to two distinct things that are easy to conflate:
h5pywithout ever constructing a DataDict.The docs should make this distinction explicit and then specify the on-disk schema precisely. This is what makes interop clear: any tool can read/write DDH5 directly, and the format (not the object) is what plottr, autoplot, and the analysis tooling all rely on.
What to document
Definition of DDH5 (HDF5 file following the DataDict storage schema) and how it relates to — but is not the same as — the DataDict object.
The on-disk schema (exact details should be pulled from
src/labcore/data/datadict_storage.py), covering at least:data) holding one HDF5 dataset per fieldaxes(independent-axis names, for dependents),unit,labelaxes)__key__double-underscore convention for internal meta keysPlus:
h5dump-style listing or a diagram)Why it matters
The format-vs-object distinction is genuinely illuminating and currently lives only implicitly in the code. Writing it down makes the storage layer much easier to reason about and to build on.