Skip to content

Add complete OMNI variable support#83

Open
Simon060899 wants to merge 4 commits into
GFZ:mainfrom
Simon060899:codex/omni-all-variables
Open

Add complete OMNI variable support#83
Simon060899 wants to merge 4 commits into
GFZ:mainfrom
Simon060899:codex/omni-all-variables

Conversation

@Simon060899

Copy link
Copy Markdown

Summary

  • add registry-driven access to all 42 one-minute and 45 five-minute OMNIWeb variables
  • add all 54 non-time OMNI2 fields, including support for historic 55-word and current 57-word hourly records
  • preserve the existing nine high-resolution and three hourly default columns and their ordering
  • add variables=None, variables="all", ordered subset selection, normalized aliases, and available_variables() metadata discovery
  • store complete cadence-valid processed schemas and upgrade partial legacy caches only when expanded reads require it
  • preserve UTC indices, source-file provenance, atomic writes, parallel monthly downloads, and focused-reader return schemas

Compatibility and robustness

Existing calls retain their previous schemas because the new argument is optional. The implementation also adds explicit validation for cadence-incompatible selections, empty or invalid selections, mixed timezone inputs, invalid ranges, malformed NASA responses, invalid time fields, corrupt caches, and unsupported hourly record widths.

Cache replacement remains atomic and uses pathlib and tempfile, so the implementation is portable across Linux, macOS, and Windows-style path handling.

Documentation

  • add a dedicated OMNI user-guide page with complete variable tables, NASA identifiers, units, fill values, cadence restrictions, API examples, cache behavior, compatibility guarantees, error guidance, and source acknowledgements
  • expand public docstrings and focused implementation comments
  • add a reviewer implementation report documenting architecture, design decisions, verification, and known boundaries
  • link the new guide from the README and Sphinx index

Verification

  • deterministic OMNI suite: 118 passed, 2 opt-in live checks skipped
  • opt-in NASA smoke tests: 2 passed, covering all 42 one-minute and all 45 five-minute fields
  • production-size live checks: complete one- and five-minute months, plus a 2024 hourly 55-word file expanded to all 54 output fields
  • full local repository suite: 586 passed, 12 skipped, with only the existing live SuperMAG endpoint test failing intermittently
  • full local suite excluding that exact external test: 586 passed, 12 skipped, 1 deselected
  • Ruff, formatting, byte-compilation, and Sphinx documentation builds passed
  • REUSE compliance passed

The manually dispatched Linux/macOS Python 3.11–3.14 matrix established that every OMNI test passes on every requested combination. Seven jobs failed only the existing live SuperMAG test (zero or one of two requested days returned); macOS/Python 3.11 received both responses and passed completely. The Ty job reports the same two RBMDataSet diagnostics reproduced on untouched upstream/main; neither is in a file changed by this PR.

Reviewer entry points

  • swvo/io/omni/variables.py — authoritative variable metadata and selection rules
  • swvo/io/omni/omni_high_res.py — complete OMNIWeb request, parsing, and monthly cache-upgrade flow
  • swvo/io/omni/omni_low_res.py — version-tolerant OMNI2 parsing and yearly cache-upgrade flow
  • docs/omni_variables.rst — complete public guide
  • docs/omni_implementation_report.md — design and verification report

@sahiljhawar

Copy link
Copy Markdown
Collaborator

@Simon060899 in child classes which are inheriting from OMNI, I think tz enforcement and time inequality assertion can be removed since parent class already have them.

A similar refactoring should be performed for DSTOMNI to make things consistent.

let me know if you need help

@sahiljhawar

Copy link
Copy Markdown
Collaborator

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants