Performance tune for prepare_insar_hdf5, resample_slc_v2, and rubbersheet - #358
Open
lijun99 wants to merge 3 commits into
Open
Performance tune for prepare_insar_hdf5, resample_slc_v2, and rubbersheet#358lijun99 wants to merge 3 commits into
lijun99 wants to merge 3 commits into
Conversation
resample_secondary_rslc_onto_reference() read the secondary RSLC via RSLC.getSlcDatasetAsNativeComplex(), which opens the file with h5py's default chunk cache instead of one sized for the dataset's own chunking. Open the secondary RSLC directly via HDF5OptimizedReader instead, matching the pattern already used in unwrap.py, crossmul.py, and other workflows. This cuts the resample step's SLC I/O read time by roughly 5x.
Replace the per-pixel Python loop (and its per-pixel calls into SubSwaths.get_sample_sub_swath) with vectorized numpy operations, reading offset raster rows only as needed to keep memory bounded. Tests show ~3.5x speedup on the prepare_insar_hdf5 step, and output verified bit-for-bit identical to the original.
0.1 is far tighter than the standard w-test threshold, causing excessive outlier-removal iterations without a corresponding accuracy benefit. 3.29 (alpha=0.001) is the conventional geodetic data-snooping critical value and matches ISCE2 precedent; validated against full-scene RUNW output with negligible impact on valid pixels.
lijun99
force-pushed
the
performance_tune
branch
from
August 17, 2026 23:32
3b83227 to
b48c793
Compare
Contributor
Author
|
The test failures seem to be pre-existing. Please take a look at this PR, @hfattahi @bhawkins @xhuang-jpl. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR aims to speed up three modules in insar workflows, prepare_insar_hdf5, resample_slc_v2, and rubbersheet. I put them together for easier testing. Please let me know if separate PRs are preferred.
prepare_insar_hdf5
Replace the per-pixel Python loop (and its per-pixel calls into SubSwaths.get_sample_sub_swath) with vectorized numpy operations, reading offset raster rows only as needed to keep memory bounded. Tests show ~3.5-10x speedup on the prepare_insar_hdf5 step, and output verified bit-for-bit identical to the original.
resample_slc_v2
Replace the legacy HDF5 reader with HDF5OptimizedReader used in unwrap.py, crossmul.py, and others. This cuts the resample step's SLC I/O read time by ~5-10x. Output is verified bit-for-bit identical to the original.
rubbersheet
Update rubbersheet polyfit critical_value from 0.1 to Baarda's convention 3.29.
0.1 (keeping only 8%) is far tighter than the standard w-test threshold, causing excessive outlier-removal iterations without a corresponding accuracy benefit. 3.29 (alpha=0.001) is the conventional geodetic data-snooping critical value. Recommended range 1.69-3.29.
Here is a test on a scene from Afar, Ethiopia,
It turns out that 0.1 removes 68.3% of good pixels for polyfit, not only makes the fitting less representative, but also leads to long iterations to finish.
In tests, choosing 3.29 leads to ~5-10x speedup for the polyfit. (Note that the rubbersheet procedure has heavy I/Os, the actual I/O performance depends on types of storage systems.) There is a small difference, ~0.2 rad across the scene, from a ramp, since the fitting parameters are slightly different.
BTW, I am also working on a gpu phase unwrapping. It shows some early success; but needs more test. Please give it a try when you have time.