Compiled Lucas-Kanade kernels (1.4.0) and Eulerian video magnification - #65
Merged
Conversation
Add EulerianMagnifier (linear EVM, Wu et al. 2012) to pyidi.postprocessing: a Laplacian pyramid + per-level temporal band-pass + linear amplification to reveal subtle, sub-pixel motion directly from raw video, before displacement identification. Qualitative visualization only, not a measurement. - Class API mirroring the IDIMethod configure() pattern, plus a functional eulerian_magnification() wrapper; both exported from postprocessing. - Ideal (FFT) or Butterworth temporal filter; Nyquist/level validation. - ROI mask option to magnify only local motion and keep the background still. - tqdm progress bars over pyramid build, temporal filtering, and reconstruction. - mp4/gif export with intensity-range display mapping. - Built on scipy.ndimage (no OpenCV dependency for the pyramid). - Tests: shape/dtype, in-band amplification, out-of-band rejection, mask gating, and input validation. - Example script magnifying an identified structural mode from a real recording.
Fuse the gradients, matrix inverse, spline evaluation and iteration into one compiled function with prange over points, reimplementing FITPACK's bispev with de Boor's algorithm (including the way it clamps coordinates outside the knot range rather than extrapolating). Also read the frame once per time step instead of once per point, and enable the @njit decorators that had been commented out on the existing helpers. numba's OpenMP threading layer cannot survive fork, so ask for a fork-safe layer at import and pin workers to one numba thread; otherwise a processes=N run after a processes=1 run dies with BrokenProcessPool. Identical results, 52-74x faster. The NumPy path stays reachable with use_compiled_kernel=False. No fastmath: it turns the divergence guards into dead code, for about 8%.
On a cold cache every worker compiled the same kernel independently. Warming it in the parent cuts a cold processes=4 run from 4.5 s to 3.1 s and avoids concurrent writes to the same cache entry (numba issue #4807). numba specializes on memory layout, so the warm-up has to match the real call: it uses two points, because with one the `previous` slice is contiguous and a different specialization gets compiled. Failures are swallowed; workers can always compile the kernels themselves.
One point that could not be tracked raised ValueError and threw away every other point with it; in a 120-point test one bad point lost all 120. Such points are now NaN from the frame at which they were lost, the analysis continues, and failed_points records the frame and reason. Applies to both failure modes and both paths; the NumPy path gains the runaway check it never had. Detection is best effort: the bound is the image size, so a point can wander to tens of pixels of nonsense without tripping it.
pyidi no longer fails to import when numba is missing: the import is guarded and exposes NUMBA_AVAILABLE, njit becomes a no-op and prange becomes range, which keeps the module importable. The kernels are then far too slow to use, so LucasKanade refuses to select them and says why. The two helpers on the NumPy path now exist in a compiled (explicit loops) and a vectorised (NumPy) flavour, bound at import, since neither is right for both cases. numba stays a required dependency: pyMRAW requires it and video_reader imports pyMRAW unconditionally, so a normal install always has it. This is what happens if it is broken, not a supported install mode.
The method has the same structure as Lucas-Kanade; only the normal equations differ, having one unknown instead of two. The spline machinery is therefore shared and only a per-point optimiser and frame driver are added. Identical results, 38-65x faster. Also reads the frame once per time step (18x on video files), reports untrackable points through failed_points instead of silently never moving them, pins workers to one numba thread, compiles before forking, and actually compiles compute_delta_numba, whose @njit decorator had been commented out. configure() takes use_compiled_kernel in place of use_numba, which did nothing.
- Test on Windows and macOS as well as Linux; the threading and multiprocessing changes were only ever exercised on Linux. - Require numba >= 0.59, for prange and config.THREADING_LAYER. - Document the compiled kernel for both methods, the compile-and-cache behaviour, NUMBA_CACHE_DIR and how untrackable points are reported. - Bump to 1.4.0: new features plus behaviour changes, not a patch.
Asking numba for a fork-safe threading layer was not enough. libgomp kills any child forked from a process that has used it, whoever started it, and numba is not the only library that pulls it in: on CI it arrived through OpenCV and every multiprocessing test died with BrokenProcessPool. It went unnoticed locally because numba resolves to TBB here and nothing maps libgomp at all. Workers now start through forkserver when libgomp is in the process. Where forking is safe it is still used, since it shares the video with the workers copy-on-write instead of pickling a copy to each of them: 7.9 s against 34.4 s for the full suite. test_threading_layer_is_fork_safe only asserted the configured value, not the layer numba resolved to, so it passed while the process was dying. It now checks the resolved layer, and a new test runs a real processes=2 analysis with OpenMP forced, which reproduces the CI failure exactly.
This is the root cause behind the previous commit. pyMRAW starts numba's thread pool at import time, and the threading layer is fixed once the pool is up, so asking for it in methods/_lk_kernels.py was always too late: pyidi imports pyMRAW first. Left at the default it resolves to TBB where that is installed and OpenMP where it is not, which is why this only ever failed on CI. The request now happens at the top of pyidi/__init__.py, before any import that reaches numba, and falls back to setting the live config when something imported numba even earlier (a pytest plugin does exactly that). The previous commit stays as the backstop for OpenMP that arrives through some other library. test_threading_layer_is_fork_safe checked the resolved layer on every platform, which failed on Windows: numba deliberately allows OpenMP under forksafe everywhere except Linux, where GNU OpenMP is the only unsafe one. It now only constrains platforms that fork.
Nine jobs down to five: every supported Python on Linux, where the runners are cheapest, and the newest one only on Windows and macOS. Those two are here for what differs by operating system, such as the process start method and the numba threading layer, and that does not vary with the Python version. actions/checkout and actions/setup-python were on node20, which GitHub has deprecated, in both workflows. v7 of each runs on node24.
- DirectionalLucasKanade: align the (pad_y, pad_x) axis pairing of _interpolate_reference and _warm_up_kernels with _padded_slice, so asymmetric pads no longer crash every point - GUIs: scale displacement fields with np.nanmax (finite fallback when all points failed) so one lost point no longer breaks the viewers - NumPy path: guard the int cast with _displacement_is_sane so a point restored from a checkpoint with NaN stays lost, matching the compiled kernel - Multiprocessing: warn about failed points in the parent with global indices after multi() merges the worker results
Run the wavelength ramp from the coarsest band toward the finest per Wu et al. 2012, warn when every level is attenuated to zero, raise in save() without a frame rate, and make the example video path an argument with its outputs gitignored.
Ten new mutation-verified tests catch a disabled band-pass, a sign-inverted amplification and a dropped pyramid band, and cover save(), Butterworth, lambda_c, odd frame sizes and float input.
Contributor
|
Great PR @klemengit |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lucas-Kanade performance
LucasKanadeandDirectionalLucasKanadeis compiled with numba and parallelized over points: 36-89x faster than 1.3.3 with identical results (benchmarks in CHANGELOG.md).configure(use_compiled_kernel=False)selects the previous NumPy path; missing numba falls back automatically.resume_analysis, and the GUI viewers scale around NaN points.pad=(pad_y, pad_x)inDirectionalLucasKanadeno longer crashes; the 1.3.3 helper names remain importable as aliases.Eulerian video magnification
pyidi.postprocessing.EulerianMagnifier(linear EVM, Wu et al. 2012): Laplacian pyramid + temporal band-pass to visualize sub-pixel vibration directly from raw video before displacement identification. Qualitative aid only, not a measurement.lambda_cspatial attenuation runs coarse-to-fine as in the reference; 16 tests, the core assertions mutation-verified.CI