Skip to content

Performance tune for prepare_insar_hdf5, resample_slc_v2, and rubbersheet - #358

Open
lijun99 wants to merge 3 commits into
isce-framework:developfrom
lijun99:performance_tune
Open

Performance tune for prepare_insar_hdf5, resample_slc_v2, and rubbersheet#358
lijun99 wants to merge 3 commits into
isce-framework:developfrom
lijun99:performance_tune

Conversation

@lijun99

@lijun99 lijun99 commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

This PR aims to speed up three modules in insar workflows, prepare_insar_hdf5, resample_slc_v2, and rubbersheet. I put them together for easier testing. Please let me know if separate PRs are preferred.

  1. prepare_insar_hdf5
    Replace the per-pixel Python loop (and its per-pixel calls into SubSwaths.get_sample_sub_swath) with vectorized numpy operations, reading offset raster rows only as needed to keep memory bounded. Tests show ~3.5-10x speedup on the prepare_insar_hdf5 step, and output verified bit-for-bit identical to the original.

  2. resample_slc_v2
    Replace the legacy HDF5 reader with HDF5OptimizedReader used in unwrap.py, crossmul.py, and others. This cuts the resample step's SLC I/O read time by ~5-10x. Output is verified bit-for-bit identical to the original.

  3. rubbersheet
    Update rubbersheet polyfit critical_value from 0.1 to Baarda's convention 3.29.
    0.1 (keeping only 8%) is far tighter than the standard w-test threshold, causing excessive outlier-removal iterations without a corresponding accuracy benefit. 3.29 (alpha=0.001) is the conventional geodetic data-snooping critical value. Recommended range 1.69-3.29.
    Here is a test on a scene from Afar, Ethiopia,

crit_value Iterations Time % of data removed Removed points with corr_peak > 0.7
0.1 (current) 23,187 72.35s 58.0% 68.3% (mostly good matches!)
1.0 3,197 16.30s 8.0% 3.4%
2.0 3,111 15.76s 7.8% 2.8%
3.29 (Baarda's standard) 3,008 14.66s 7.5% 2.8%

It turns out that 0.1 removes 68.3% of good pixels for polyfit, not only makes the fitting less representative, but also leads to long iterations to finish.
In tests, choosing 3.29 leads to ~5-10x speedup for the polyfit. (Note that the rubbersheet procedure has heavy I/Os, the actual I/O performance depends on types of storage systems.) There is a small difference, ~0.2 rad across the scene, from a ramp, since the fitting parameters are slightly different.

BTW, I am also working on a gpu phase unwrapping. It shows some early success; but needs more test. Please give it a try when you have time.

resample_secondary_rslc_onto_reference() read the secondary RSLC via
RSLC.getSlcDatasetAsNativeComplex(), which opens the file with h5py's
default chunk cache instead of one sized for the dataset's own chunking.
Open the secondary RSLC directly via HDF5OptimizedReader instead, matching
the pattern already used in unwrap.py, crossmul.py, and other workflows.

This cuts the resample step's SLC I/O read time by roughly 5x.
Replace the per-pixel Python loop (and its per-pixel calls into
SubSwaths.get_sample_sub_swath) with vectorized numpy operations,
reading offset raster rows only as needed to keep memory bounded.
Tests show ~3.5x speedup on the prepare_insar_hdf5 step, and output
verified bit-for-bit identical to the original.
0.1 is far tighter than the standard w-test threshold, causing
excessive outlier-removal iterations without a corresponding
accuracy benefit. 3.29 (alpha=0.001) is the conventional geodetic
data-snooping critical value and matches ISCE2 precedent; validated
against full-scene RUNW output with negligible impact on valid pixels.
@lijun99

lijun99 commented Aug 17, 2026

Copy link
Copy Markdown
Contributor Author

The test failures seem to be pre-existing. Please take a look at this PR, @hfattahi @bhawkins @xhuang-jpl.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant