Skip to content

[ESSREDUCE] refactor: import ess.reduce submodules lazily - #728

Open
SimonHeybrock wants to merge 1 commit into
mainfrom
lazy-reduce-init
Open

[ESSREDUCE] refactor: import ess.reduce submodules lazily#728
SimonHeybrock wants to merge 1 commit into
mainfrom
lazy-reduce-init

Conversation

@SimonHeybrock

@SimonHeybrock SimonHeybrock commented Aug 26, 2026

Copy link
Copy Markdown
Member

ess/reduce/__init__.py eagerly imports nexus, normalization, polarization, uncertainty and unwrap, so importing any part of the package pays for all of it. Because importing a submodule always executes the parent __init__, an importer that wants one piece has no way to opt out.

lazy.attach defers each submodule to first attribute access, the pattern scippneutron already uses in its own __init__. streaming is now listed alongside the others — it was importable but missing from __all__.

Effect

Measured on this branch, python -c "<import>", best of three:

eager lazy
import ess.reduce 665 ms, 177 MB, 1342 modules 39 ms, 17 MB, 154 modules
from ess.reduce import streaming 715 ms, 177 MB, 1343 modules 244 ms, 60 MB, 676 modules
from ess.reduce import unwrap 737 ms, 176 MB 720 ms, 176 MB

Someone who wants unwrap pays what they always did. Someone who wants streaming no longer pays for scippnexus, scippneutron and — when it is installed — numba, none of which streaming touches.

Downstream savings depend on what else the application imports. ESSlivedata's dashboard imports ess.reduce.streaming for the accumulators in its Kafka message adapter and never interpolates a lookup table; its total import time is unchanged, because it loads panel, bokeh, holoviews and scipp regardless, but its resident set drops from 287 to 208 MB. That case is what prompted this: scipp/esslivedata#1253 makes numba a hard dependency so backend services stop silently falling back to the SciPy interpolator, which would otherwise put ~68 MB of numba/llvmlite into every dashboard process for a module it never calls.

Relation to #707

#707 moves the numba import inside _get_interpolator_class, which removes the forced numba import from this path independently of this PR. The two overlap but neither replaces the other. For from ess.reduce import streaming, best of seven:

time maxrss modules
today 734 ms 177 MB 1343
with #707 563 ms 110 MB 1078
this PR 211 ms 60 MB 676

The #707 row is approximated by making numba unimportable, so that path falls back to SciPy; #707 itself imports neither interpolator at module level, which would make its row marginally better still. Whichever lands first, the other keeps most of its effect: #707 accounts for the numba share, this PR for the nexus -> scippnexus/scippneutron tree.

Notes

  • __init__.pyi keeps the submodules visible to type checkers, as lazy.attach alone hides them.
  • lazy_loader is added to dependencies. It was already installed everywhere via scippneutron, but it is now a direct import.
  • pixi.lock is untouched: lazy_loader 0.5 already resolves into every environment, and CI runs frozen: true. Regenerate if your convention requires it regardless.
  • Behaviour is unchanged for callers: dir(), attribute access on ess.reduce, and from ess.reduce import X all work as before. Deferring is safe because these submodule __init__s are pure re-export aggregations with no import-time registration.

Test plan

  • pytest tests for essreduce: 912 passed
  • ESSlivedata's suite against this branch: 4768 passed
  • ruff check / ruff format clean on the changed files

@github-actions github-actions Bot added the essreduce Issues for essreduce. label Aug 26, 2026
@github-actions github-actions Bot changed the title refactor: import ess.reduce submodules lazily [ESSREDUCE] refactor: import ess.reduce submodules lazily Aug 26, 2026
ess/reduce/__init__.py eagerly imported nexus, normalization, polarization,
uncertainty and unwrap, so importing any part of the package paid for all of
it. Because importing a submodule always executes the parent __init__, an
importer wanting one piece had no way to opt out.

lazy.attach defers each submodule to first attribute access, the pattern
scippneutron already uses. A stub keeps the submodules visible to type
checkers. streaming is now listed alongside the others; it was importable but
missing from __all__.

    import ess.reduce                  665 ms -> 39 ms, 177 MB -> 17 MB
    from ess.reduce import streaming   715 ms -> 244 ms, 177 MB -> 60 MB
    from ess.reduce import unwrap      unchanged

Importers that want unwrap pay what they always did. Importers that want
streaming no longer pay for scippnexus, scippneutron and, when installed,
numba, none of which streaming touches.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

essreduce Issues for essreduce.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant