Improved ORSO file parser

[rozyczko]

Assessment of easyreflectometry's ORSO file support against the
ORSO file format specification
and the ORSO simple model language,
compared with refnx, refl1d, and GenX.

reflectometry-lib reads the ORSO .ort format properly via orsopy and its reading support is on par with refnx and GenX and ahead of refl1d,
including the "simple model" language. Where it falls behind the other codes is
writing: it cannot export .ort files at all, and it has no support for the binary
.orb format.

What the library supports today

• Data columns: reads the four mandated columns Qz, R, sR, sQz; errors are stored
  as variances and sQz get passed to the Pointwise resolution function
  (falling back to 5% FWHM when absent). This matches how
  refl1d/refnx treat dq.
• Multiple datasets per file : yes; each # data_set: block becomes a separate
  R_<name> / Qz_<name> pair, and Project.load_all_experiments_from_file converts
  them into experiments.
• Header metadata: the full ORSO header is preserved as a dict; two things are
  actively used: the experiment title, and the polarization channel (pp/pm/mp/mm).
• ORSO model language ("simple model") : supported for reading
  including SLD unit conversion and density-only materials. Not handled
  yet: globals overrides (length unit, default roughness) and magnetic fields in
  the model language.

Comparison with other codes

Capability	reflectometry-lib	refnx	refl1d	GenX
Read .ort	✅ (orsopy)	✅	✅ (load4/parse_orso)	✅ (auto loader)
Write .ort	❌	✅	not verified (no .ort writer found in load4.py; probe save targets refl1d's own format)	✅ (full export incl. fit results)
Read binary .orb	❌	✅ (load_nexus fallback)	✅ (load_nexus)	unverified (likely ❌; GenX uses its own .hgx HDF)
Convert 1/nm Q to internal unit	❌ (unit stored, values passed raw)	✅ (_data[0] /= 10.0)	unverified	—
Honour value_is: FWHM on sQz	❌ (assumes σ)	❌ (assumes σ, ×2.3548 to FWHM)	✅ (FWHM2sigma)	—
Model language → model	✅ (with caveats)	✅ (slabs only, setup_analysis)	❌	✅ (read + write)
Write model language back	❌	✅ (update_model)	❌	✅ (analysis section + params)
Polarization from header	partial (dataset 0 only; cannot see per-dataset overrides)	partial	✅ (per dataset, incl. po/mo coercion)	✅
Instrument settings from header	❌	partial	✅ (angle/wavelength)	✅

Reading is good, arguably the polarization-header handling is better than
refnx's. The clear parity gaps are export (refnx round-trips refined parameters
back into the .ort via OrsoDataset.update_model(); GenX exports the entire fit:
script, parameters, uncertainties, and an ORSO-model-language description) and
.orb (refnx reads/writes it; orsopy 1.2.2 already ships
load_nexus/save_nexus, so method exists but is unused). Model.as_orso()
in model/model.py:282-284 is currently a stub returning the EasyScience dict,
not ORSO.

Reader weaknesses found during the audit

1. Positional column reading : orso_utils.py:223-246 indexes data[:, 0..3]
   unconditionally without validating column names against o.info.columns. A
   spec-valid 3-column file (sQz is only recommended) raises IndexError, which
   load() silently swallows and re-parses as plain text. The file loads but all
   ORSO metadata (including polarization) is lost. A file with reordered columns
   would be silently mis-read.
2. Polarized multi-dataset files rejected : Project.load_polarized_experiment
   (project.py:770-774) requires one file per spin channel, so a single .ort
   carrying all four channels as datasets (the format's intended way to store
   polarized data) cannot be imported as a polarized experiment, even though each
   dataset's header names its channel.
3. orsopy is unpinned in pyproject.toml:30 : worth at least a lower bound
   (≥1.2) since the model-language API surface changed across versions.
4. Minor : Project.load_orso_file stores a raw scipp DataGroup where other
   paths store DataSet1D, and instrument settings (wavelength, angles, probe) are
   never propagated from the header.

Update tasks (revised priority)

#	Task	Effort	Impact
1	Pin orsopy >= 1.2 in pyproject.toml	Trivial	Correctness for fresh installs
2	Harden the reader: name-based column validation via info.columns; honour value_is (FWHM→σ); all-nan sQz → leave xe empty so _apply_resolution_function falls back to 5% FWHM; partial-nan sQz needs per-point handling (nan-mask and interpolate over valid points, with documented semantics) — not just the empty-column case; warn on (or remove) the silent resolve_to_layers() → resolve_stack() fallback; decide the Pointwise sigma-vs-variance serialization contract (saved projects round-trip sQz_data_points as variances — changing it silently breaks them; add a migration note); stop swallowing ORSO parse failures. Discriminator: the ORSO banner line (# # ORSO reflectivity data file), not the .ort extension — fall back to _load_txt only when the banner is absent, raise on a bannered file that fails to parse. Review the fallback policy of the three other call sites that tolerate ORSO failure by design: count_datasets_in_file, load_all_experiments_from_file, _apply_experiment_metadata.	Small–Medium	High — ends silent metadata loss and mis-smearing
3	Project.load_orso_file: deletion is the default — it is unused by the main load_new_experiment path and is a documented footgun. Before deleting, check external callers (the GUI app consumes Project's public API); if one exists, deprecate and reimplement as a thin wrapper over load_new_experiment (DataSet1D + same title/resolution path). Add a model-bearing test through whichever path survives using Ni_example.ort — the existing test uses example.ort, a 0.1-standard file with no sample.model (legacy dimension: columns), so it exercises default_model(), not the ORSO path.	Small	Medium
4	Preserve the original sample.model (keep globals, materials, sub_stacks); honour length_unit (default nm) and default roughness. Mechanism for repeats: resolve_to_layers() cannot preserve them — orsopy's SubStack.resolve_to_layers() ends with return layers * self.repetitions (verified in 1.2.2). Use resolve_to_blocks() (keeps SubStackType objects intact) and map SubStack with repetitions > 1 → ERL RepeatingMultilayer explicitly; keep density materials unflattened.	Medium	High — prerequisite for both correct reads and any exporter
5	Polarized import from a single multi-dataset .ort: per-dataset header classification; defined error on ambiguity (two pp, mixed unpolarized); do not coerce po/mo/op/om; vector is explicitly out of scope (a declared-but-unmapped header value already suppresses the filename fallback and yields None — document, don't change); decide API ownership vs load_all_experiments_from_file (multi-spin vs multi-angle look identical at data_set: level); decide per-channel vs shared resolution before task 6, which depends on the answer; fix the repeated re-parsing (count_datasets_in_file + load_as_dataset + _apply_experiment_metadata each reload the file — up to 3-4 full parses per channel); add a real multi-channel fixture	Medium	High
6	Slab-only .ort exporter: Sample → SampleModel (superphase | layers | subphase), SLD back-conversion to Å⁻², emit Qz/R/sR/sQz as sigma — i.e. sqrt(ye)/sqrt(xe), since DataSet1D stores variances; reuse the preserved orso_header (data_source/reduction) rather than synthesizing one; save_orso; replace the Model.as_orso() stub (verified never called in src/). Explicitly without a GenX-style analysis block (separate product decision). Export semantics must be decided: one .ort per experiment vs one file with N data_set: blocks — the reader critique (weakness #3) demands multi-dataset support, so the writer should be able to produce it too, at least for polarized experiments. Depends on tasks 4 and 5's resolution decision, or repeats round-trip as flattened slabs.	Medium	High — refnx-level write support
7	.orb read via orsopy load_nexus (catch-up with refnx and refl1d); requires h5py — add as an [orb] extra or document the orsopy nexus extra. .orb write is task 6 with another extension.	Small (read)	Medium
7a	Add the five missing fixture classes to tests/_static/: nan-filled sQz, value_is: FWHM, 1/nm Q units, multi-channel polarized .ort, .orb. Each lands with the task that consumes it (2, 2, 2, 5, 7) but is listed here so the test debt is visible.	Small	Enables tasks 2/5/7 verification
8	Deferred pending product decisions: magnetism from model language (gate on calculator_supports_magnetism), instrument-settings propagation (needs a home — ERL has no first-class instrument object), GenX-style analysis: export. The ORSO-target-version decision should be resolved first among these: orsopy hard-codes ORSO_VERSION = "1.2" and will stamp every written file with it regardless of what ERL documents.	—	—

Sources

• ORSO simple model spec
• ORSO file format specification
• refnx ORSO interchange docs
• GenX ORSO integration
• orsopy
• refl1d releases