[ESSREDUCE] refactor: import ess.reduce submodules lazily - #728
Open
SimonHeybrock wants to merge 1 commit into
Open
[ESSREDUCE] refactor: import ess.reduce submodules lazily#728SimonHeybrock wants to merge 1 commit into
SimonHeybrock wants to merge 1 commit into
Conversation
ess/reduce/__init__.py eagerly imported nexus, normalization, polarization,
uncertainty and unwrap, so importing any part of the package paid for all of
it. Because importing a submodule always executes the parent __init__, an
importer wanting one piece had no way to opt out.
lazy.attach defers each submodule to first attribute access, the pattern
scippneutron already uses. A stub keeps the submodules visible to type
checkers. streaming is now listed alongside the others; it was importable but
missing from __all__.
import ess.reduce 665 ms -> 39 ms, 177 MB -> 17 MB
from ess.reduce import streaming 715 ms -> 244 ms, 177 MB -> 60 MB
from ess.reduce import unwrap unchanged
Importers that want unwrap pay what they always did. Importers that want
streaming no longer pay for scippnexus, scippneutron and, when installed,
numba, none of which streaming touches.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
SimonHeybrock
force-pushed
the
lazy-reduce-init
branch
from
August 26, 2026 05:06
fb334c4 to
a9d5c5b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ess/reduce/__init__.pyeagerly importsnexus,normalization,polarization,uncertaintyandunwrap, so importing any part of the package pays for all of it. Because importing a submodule always executes the parent__init__, an importer that wants one piece has no way to opt out.lazy.attachdefers each submodule to first attribute access, the pattern scippneutron already uses in its own__init__.streamingis now listed alongside the others — it was importable but missing from__all__.Effect
Measured on this branch,
python -c "<import>", best of three:import ess.reducefrom ess.reduce import streamingfrom ess.reduce import unwrapSomeone who wants
unwrappays what they always did. Someone who wantsstreamingno longer pays for scippnexus, scippneutron and — when it is installed — numba, none of whichstreamingtouches.Downstream savings depend on what else the application imports. ESSlivedata's dashboard imports
ess.reduce.streamingfor the accumulators in its Kafka message adapter and never interpolates a lookup table; its total import time is unchanged, because it loads panel, bokeh, holoviews and scipp regardless, but its resident set drops from 287 to 208 MB. That case is what prompted this: scipp/esslivedata#1253 makesnumbaa hard dependency so backend services stop silently falling back to the SciPy interpolator, which would otherwise put ~68 MB of numba/llvmlite into every dashboard process for a module it never calls.Relation to #707
#707 moves the numba import inside
_get_interpolator_class, which removes the forced numba import from this path independently of this PR. The two overlap but neither replaces the other. Forfrom ess.reduce import streaming, best of seven:The #707 row is approximated by making
numbaunimportable, so that path falls back to SciPy; #707 itself imports neither interpolator at module level, which would make its row marginally better still. Whichever lands first, the other keeps most of its effect: #707 accounts for the numba share, this PR for thenexus-> scippnexus/scippneutron tree.Notes
__init__.pyikeeps the submodules visible to type checkers, aslazy.attachalone hides them.lazy_loaderis added todependencies. It was already installed everywhere via scippneutron, but it is now a direct import.pixi.lockis untouched:lazy_loader0.5 already resolves into every environment, and CI runsfrozen: true. Regenerate if your convention requires it regardless.dir(), attribute access oness.reduce, andfrom ess.reduce import Xall work as before. Deferring is safe because these submodule__init__s are pure re-export aggregations with no import-time registration.Test plan
pytest testsfor essreduce: 912 passedruff check/ruff formatclean on the changed files