Move panda flyer - #37
Thomas Hopkins (thopkins32) wants to merge 8 commits into
Conversation
…-collection Move the PandABox fly-scan flyer and its two HDF5 export helpers out of the IPython-profile startup scripts (hxn-profile-collection/startup/93-scanplan-panda.py, 95-xspress3-panda.py) into hxntools so HXNFlyerPanda is importable and testable standalone, independent of the profile's shared IPython namespace. - HXNFlyerPanda gains an explicit constructor: large_file_directory_root/ write_path/read_path are now required keyword-only arguments (previously class attributes resolved from profile-only globals at class-definition time), live_plot_callback replaces the direct panda_live_plot.update_plot(...) call (hxntools has no beamline live-plotting dependency of its own), and n_scaler_mca is a constructor argument (previously the profile global n_scaler_mca). - kickoff()/complete()/describe_collect()/collect()/collect_asset_docs() are otherwise unchanged from the profile version.
describe_collect() declared the scaler ("sclr1_ch*") and Xspress3-ROI
("Det{n}_<element>") keys' shape using the raw, position_supersample-
oversampled PandA gate count (self.frame_per_point) instead of the actual
number of scan points. ExportSISDataPanda.export()/ExportXpsROI.export()
already collapse that oversampling before writing their HDF5 files (both
divide by position_supersample), so the declared shape was always
position_supersample times too large -- exactly 10x under every 2D/1D PandA
plan's default position_supersample=10.
- describe_collect(): num_scan_points = frame_per_point // position_supersample.
- kickoff(): matching frame_per_point // position_supersample in both the SIS
and ROI resource_factory() resource_kwargs, so Tiled's structure validator
reshapes the on-disk array to the same per-row length as the descriptor.
- Add tests/test_panda_flyer.py: drives the real HXNFlyerPanda through
kickoff/complete/collect against a real, in-process Tiled server and
asserts the declared and on-disk array shapes match the true point count,
for position_supersample in {1, 10}.
- Add the [project.optional-dependencies].test extra (pytest, tiled[server],
bluesky-tiled-plugins) needed to run the new test.
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Acquisition correctness issues and missing CI/lockfile integration must be addressed.
Get a fresh assessment by requesting another Copilot review.
Review effort: Balanced
Findings: 5
Open (7)
uv.lock is stale and missing declared development dependencies · New MCA length check truncates short arrays instead of padding · New Timeout uses wrong units and reports partial acquisition as success · New Event timestamps incorrectly use datum IDs instead of numeric times · New External dataset descriptors omit the float32 dtype · New CI does not run pytest regression tests · New Command requests undefined test extra instead of the dev group · New
What changed in this PR
Ports the PandA flyer into hxntools and corrects oversampled scaler/ROI descriptor shapes.
Changes:
- Adds
HXNFlyerPandaand HDF5 exporters. - Adds a Tiled-backed shape regression test.
- Adds test dependencies.
| File | Description |
|---|---|
src/hxntools/panda_flyer.py |
Adds flyer acquisition and descriptor logic. |
tests/test_panda_flyer.py |
Tests declared and stored array shapes. |
pyproject.toml |
Adds integration-test dependencies. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
CI previously only ran pre-commit (ruff) and a package-import smoke check (regular/editable install + first few discovered submodules); no job ever invoked pytest, so tests/test_panda_flyer.py was never exercised in CI. - Add a tests job (matrix: Python 3.11/3.12) that runs 'uv sync --locked --group dev' + 'pytest tests -v'. - Add doct/pims to [project.dependencies]: hxntools.handlers unconditionally imports timepix.py, which needs databroker.assets.handlers, which needs doct (databroker.utils) and pims (databroker.assets.readers.spe) -- both only declared as optional extras of databroker, not pulled in by our bare 'databroker' dependency. Without this, pytest can't even collect tests/test_panda_flyer.py (ModuleNotFoundError: No module named 'doct'), since it imports hxntools.panda_flyer -> hxntools.handlers.rasmi2 -> hxntools.handlers (package __init__, which eagerly imports every handler submodule including timepix). - Regenerate uv.lock (it was already stale relative to pyproject.toml before this change, from the prior merge of upstream main's packaging migration; 'uv lock --check' failed until this commit).
…ntain permissions' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The flyer has data-integrity, timeout, multi-detector, timestamp, and CI-lint issues that should be fixed first.
Get a fresh assessment by requesting another Copilot review.
Review effort: Balanced
Findings: 2
Open (3)
Resolved since last review (7)
External dataset descriptors omit the float32 dtype Event timestamps incorrectly use datum IDs instead of numeric times Timeout uses wrong units and reports partial acquisition as success MCA length check truncates short arrays instead of padding uv.lock is stale and missing declared development dependencies CI does not run pytest regression tests Command requests undefined test extra instead of the dev group
_spec() left dtype unset for the scaler ('sclr1_ch*') and Xspress3-ROI
('Det{n}_<element>') external keys, even though it's known and fixed:
ExportSISDataPanda/ExportXpsROI both hardcode a 32-bit float HDF5 dataset
(dtype="f") for every channel/ROI. Left unset, bluesky_tiled_plugins fell
back to the JSON-schema default (float64) and rejected the mismatched
on-disk dtype at read time -- worked around in the test via a 'descriptor'
normalizer patch (_fix_external_dtype).
Unlike the datum_kwargs naming (column/det_elem/field, still required by the
registered legacy SISHDF5Handler/ROIHDF5Handler/PandAHandlerHDF5 __call__
signatures in hxntools/handlers/), nothing depends on this dtype being
*unset* -- it's purely additive metadata, safe to declare directly. Add
dtype_str="<f4" to _spec() and delete the now-unnecessary test patch.
|
Waiting to test at beamline before we merge. |
|
Dan Henriksen (@dihenriksen) good to merge? |



Summary
Primarily does two things:
I made these two changes as separate commits so you can clearly see where the shape fix comes into play.
Motivation / Context
While doing an audit of descriptor data across NSLS-II beamlines, I discovered that there are 240 data keys that describe their data shape incorrectly, all stemming from the HXNFlyerPanda device. In this case, the declared shape was simply 10x what it was supposed to be.
I intentionally port the device definition here so we can write proper tests against this device, such that no reversion can easily occur. I intentionally left the device code unchanged, although it may be prudent to adopt the ophyd-async version in the near future and remove this code entirely. How to handle SIS data and Xps3 ROI data with the ophyd-async device is an open question...I think we will have to rewrite how this is done as well to make it compatible.
Evidence
Det{1,2,3}_<element>[_L|_M]— per-detector element maps (e.g.Det1_Fedeclared[49000], stored[4900])Det4_*— declared[64000], stored[6400];Det4_Pt_L[100]→[10]sclr1_ch1…sclr1_ch16— declared[49000], stored[4900]xspress3_channel{1,2,3}_rois_roi01…roi16— declared[1000], stored[100]