Skip to content

Move panda flyer - #37

Open
Thomas Hopkins (thopkins32) wants to merge 8 commits into
mainfrom
move-panda-flyer
Open

Thomas Hopkins (thopkins32) wants to merge 8 commits into
mainfrom
move-panda-flyer

Conversation

@thopkins32

@thopkins32 Thomas Hopkins (thopkins32) commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Primarily does two things:

  • Ports the HXNFlyerPanda device + dependencies from hxn-profile-collection
  • Fixes the descriptor shapes declared by the device

I made these two changes as separate commits so you can clearly see where the shape fix comes into play.

Motivation / Context

While doing an audit of descriptor data across NSLS-II beamlines, I discovered that there are 240 data keys that describe their data shape incorrectly, all stemming from the HXNFlyerPanda device. In this case, the declared shape was simply 10x what it was supposed to be.

I intentionally port the device definition here so we can write proper tests against this device, such that no reversion can easily occur. I intentionally left the device code unchanged, although it may be prudent to adopt the ophyd-async version in the near future and remove this code entirely. How to handle SIS data and Xps3 ROI data with the ophyd-async device is an open question...I think we will have to rewrite how this is done as well to make it compatible.

Evidence

  • Keys (all in the fly-scan stream, n_events = 1):
    • Det{1,2,3}_<element>[_L|_M] — per-detector element maps (e.g. Det1_Fe declared [49000], stored [4900])
    • Det4_* — declared [64000], stored [6400]; Det4_Pt_L [100] → [10]
    • sclr1_ch1 … sclr1_ch16 — declared [49000], stored [4900]
    • xspress3_channel{1,2,3}_rois_roi01 … roi16 — declared [1000], stored [100]
  • Stored lengths are scan pixel counts (4900 = 70², 3136 = 56², 6400 = 80², 2500 = 50², 10000 = 100², 22500 = 150², 200 = 20×10 …). Declared = pixels × 10.

…-collection

Move the PandABox fly-scan flyer and its two HDF5 export helpers out of the
IPython-profile startup scripts (hxn-profile-collection/startup/93-scanplan-panda.py,
95-xspress3-panda.py) into hxntools so HXNFlyerPanda is importable and
testable standalone, independent of the profile's shared IPython namespace.

- HXNFlyerPanda gains an explicit constructor: large_file_directory_root/
  write_path/read_path are now required keyword-only arguments (previously
  class attributes resolved from profile-only globals at class-definition
  time), live_plot_callback replaces the direct panda_live_plot.update_plot(...)
  call (hxntools has no beamline live-plotting dependency of its own), and
  n_scaler_mca is a constructor argument (previously the profile global
  n_scaler_mca).
- kickoff()/complete()/describe_collect()/collect()/collect_asset_docs() are
  otherwise unchanged from the profile version.
describe_collect() declared the scaler ("sclr1_ch*") and Xspress3-ROI
("Det{n}_<element>") keys' shape using the raw, position_supersample-
oversampled PandA gate count (self.frame_per_point) instead of the actual
number of scan points. ExportSISDataPanda.export()/ExportXpsROI.export()
already collapse that oversampling before writing their HDF5 files (both
divide by position_supersample), so the declared shape was always
position_supersample times too large -- exactly 10x under every 2D/1D PandA
plan's default position_supersample=10.

- describe_collect(): num_scan_points = frame_per_point // position_supersample.
- kickoff(): matching frame_per_point // position_supersample in both the SIS
  and ROI resource_factory() resource_kwargs, so Tiled's structure validator
  reshapes the on-disk array to the same per-row length as the descriptor.
- Add tests/test_panda_flyer.py: drives the real HXNFlyerPanda through
  kickoff/complete/collect against a real, in-process Tiled server and
  asserts the declared and on-disk array shapes match the true point count,
  for position_supersample in {1, 10}.
- Add the [project.optional-dependencies].test extra (pytest, tiled[server],
  bluesky-tiled-plugins) needed to run the new test.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Acquisition correctness issues and missing CI/lockfile integration must be addressed.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 5 High severity · 1 Medium severity · 1 Low severity

Open (7)
What changed in this PR

Ports the PandA flyer into hxntools and corrects oversampled scaler/ROI descriptor shapes.

Changes:

  • Adds HXNFlyerPanda and HDF5 exporters.
  • Adds a Tiled-backed shape regression test.
  • Adds test dependencies.
File Description
src/​hxntools/​panda_flyer.py Adds flyer acquisition and descriptor logic.
tests/​test_panda_flyer.py Tests declared and stored array shapes.
pyproject.toml Adds integration-test dependencies.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread pyproject.toml
Comment thread src/hxntools/panda_flyer.py
Comment thread src/hxntools/panda_flyer.py
Comment thread src/hxntools/panda_flyer.py
Comment thread src/hxntools/panda_flyer.py
Comment thread tests/test_panda_flyer.py
Comment thread tests/test_panda_flyer.py Outdated
CI previously only ran pre-commit (ruff) and a package-import smoke check
(regular/editable install + first few discovered submodules); no job ever
invoked pytest, so tests/test_panda_flyer.py was never exercised in CI.

- Add a tests job (matrix: Python 3.11/3.12) that runs
  'uv sync --locked --group dev' + 'pytest tests -v'.
- Add doct/pims to [project.dependencies]: hxntools.handlers unconditionally
  imports timepix.py, which needs databroker.assets.handlers, which needs
  doct (databroker.utils) and pims (databroker.assets.readers.spe) -- both
  only declared as optional extras of databroker, not pulled in by our bare
  'databroker' dependency. Without this, pytest can't even collect
  tests/test_panda_flyer.py (ModuleNotFoundError: No module named 'doct'),
  since it imports hxntools.panda_flyer -> hxntools.handlers.rasmi2 ->
  hxntools.handlers (package __init__, which eagerly imports every handler
  submodule including timepix).
- Regenerate uv.lock (it was already stale relative to pyproject.toml before
  this change, from the prior merge of upstream main's packaging migration;
  'uv lock --check' failed until this commit).
Comment thread .github/workflows/ci.yml Fixed
…ntain permissions'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread src/hxntools/panda_flyer.py
Comment thread src/hxntools/panda_flyer.py
Comment thread src/hxntools/panda_flyer.py
_spec() left dtype unset for the scaler ('sclr1_ch*') and Xspress3-ROI
('Det{n}_<element>') external keys, even though it's known and fixed:
ExportSISDataPanda/ExportXpsROI both hardcode a 32-bit float HDF5 dataset
(dtype="f") for every channel/ROI. Left unset, bluesky_tiled_plugins fell
back to the JSON-schema default (float64) and rejected the mismatched
on-disk dtype at read time -- worked around in the test via a 'descriptor'
normalizer patch (_fix_external_dtype).

Unlike the datum_kwargs naming (column/det_elem/field, still required by the
registered legacy SISHDF5Handler/ROIHDF5Handler/PandAHandlerHDF5 __call__
signatures in hxntools/handlers/), nothing depends on this dtype being
*unset* -- it's purely additive metadata, safe to declare directly. Add
dtype_str="<f4" to _spec() and delete the now-unnecessary test patch.
@thopkins32

Copy link
Copy Markdown
Contributor Author

Waiting to test at beamline before we merge.

@thopkins32

Copy link
Copy Markdown
Contributor Author

Dan Henriksen (@dihenriksen) good to merge?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants