Feature/noisepy eval - #3
Merged
Merged
Conversation
… trim - parse_mseed: libmseed MS3TraceList fast path assembles contiguous segments in C (bench.mseed: 217 ms -> 37 ms, now faster than obspy.read at 42 ms; bit-identical output); per-record fallback kept for malformed v2 headers; collect_flags now opt-in; encoding taken from each segment's first record; num_segments now counts true segments (was: miniSEED records) - TraceBundle: segments(), to_dict(fill_value=) with true sample placement (validated exactly equal to obspy merge(method=1, fill_value=0) on gapped + overlap fixtures; later-segment-overwrites), trim(start_ns, end_ns), overlaps(); gappy default to_dict() warns - bundle_to_xarray routes through the same placement (NaN fill default) so the time coordinate is honest across gaps - S3OpenClient/S3AuthClient/FDSNMultiClient join chunks in submission order (was as_completed: non-deterministic byte order); bulk results sorted to submission order after live progress - SeisfetchClient.get_numpy/get_xarray: sample-precise window trim by default (trim=False restores whole-object behavior) - pymseed pinned >=0.6,<0.9; private-API sid fallback isolated+guarded - new fixtures (<25 KB each) + generator; 16 new tests Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…adapter, precision bank - benchmarks/: runner.py writes results/<tag>_<date>.json; suites.py (parse, cold_import, memory, footprint, s3_pull --live); render_results regenerates RESULTS.md. Committed m1-native results: parse 33.6 ms vs obspy 38.3 ms on the 11 MB day file (was 217 ms before Phase 1), tracemalloc peak 27.7 vs 52.0 MB, installed 80 vs 311 MB. Finding: cold import seisfetch 0.27 s vs obspy 0.13 s (top-level boto3 import). - seisfetch/contrib/noisepy_adapter.py: numpy/scipy ports of the obspy ops NoisePy's preprocess_raw uses at rm_resp=NO (check_sample_gaps, hann taper with in-place dtype semantics, merge(method=1,fill=0) via to_dict placement, obspy-recipe Fourier resample exact to the last ulp, trim(pad=True)), NpChannelData duck type, SeisfetchS3RawStore (URLs owned by seisfetch.s3), preprocess_raw_np full chain; [noisepy] extra (scipy only, no obspy) - to_dict(fill_value=): keep data dtype when fill fits (0 in float32 stays float32 — obspy merge semantics); promote to float64 otherwise - tests/precision/: parse identity vs obspy per segment across all encodings+gap topologies (obspy contiguous-run splits and 32 ns UTCDateTime float rounding normalized in the test harness); per-op equivalence EXACT for taper/merge/bandpass/resample/trim; full preprocess chain EXACT vs the obspy chain on gapped+float32 fixtures Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… report - benchmarks/noisepy_eval/run_ccf_eval.py: identical SCEDC bytes through (A) obspy.read + noisepy preprocess_raw and (B) seisfetch parse + adapter ports, then noisepy's own compute_fft/correlate. Result on CI.PASC 2022-01-02 (EN/EZ/NZ/ZZ daily CCFs): max abs diff 0.0 — bit-identical — and identical stretching-dv/v grid cells; results JSON committed. Integration-marked wrapper in tests/precision/. - adapter: segment_interpolate_np port; sub-sample alignment wired inside the resample branch only (mirrors noisepy placement) - seisfetch/__init__: transport imports (boto3/httpx) now lazy via PEP 562 — cold import 0.27 -> 0.10 s, 2.4x faster than obspy - benchmarks/docker/: Dockerfile.bench + run_matrix.sh (fargate-class 2cpu/4g, lambda-class 1g and 512m). Finding: obspy publishes no linux/aarch64 wheels, so arm64 (Graviton) installs must compile from source; seisfetch installs from wheels - docs/noisepy-obspy-replacement-report.md: full evaluation report; machine-matrix table pending the container runs Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fargate-class (2cpu/4g), lambda-1g, lambda-512m runs committed. Both stacks parse a day file in 156 MB RSS -> 512 MB Lambda feasible; cold import seisfetch 1.5-1.7x faster everywhere; parse loses its native edge under CPU throttling (1.3-1.6x slower than obspy; documented honestly with the pymseed bulk-accessor lever). Verdict: justified — bit-identical CCFs, 80 vs 311 MB footprint, no aarch64 obspy wheels vs pure-wheel seisfetch install. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR adds an evaluation-grade, ObsPy-free data-path for running NoisePy-style preprocessing on seisfetch-parsed miniSEED, alongside determinism fixes (byte/result ordering) and expanded precision/benchmark coverage to validate bit-level equivalence against ObsPy/NoisePy.
Changes:
- Extend
parse_mseedto emit true continuous segments (fast path via libmseed trace list + guarded per-record fallback) and add gap/overlap/trim + gap-awareto_dict(fill_value=...). - Make multi-source fetch/join outputs deterministic (submission order) across S3, FDSN multi-client, and bulk helpers; add client-level window trimming default.
- Add NoisePy adapter ports + precision/integration tests, benchmarks, and an evaluation report; introduce lazy top-level imports and a CI workflow.
Reviewed changes
Copilot reviewed 29 out of 37 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_segments_api.py | Validates new segment/gap/overlap/trim + to_dict(fill_value=...) semantics. |
| tests/test_s3_determinism.py | Tests deterministic multi-day S3 concatenation order and client-side trimming (moto). |
| tests/precision/test_preprocess_equivalence.py | Exact-equality checks for numpy/scipy ports vs ObsPy operations and full chain. |
| tests/precision/test_parse_identity.py | Asserts bit-identical decode vs ObsPy per segment across fixtures/encodings. |
| tests/precision/test_ccf_equivalence.py | Integration-marked end-to-end CCF equivalence harness runner check. |
| tests/precision/init.py | Declares precision test package. |
| tests/fixtures/make_fixtures.py | Script to generate committed miniSEED fixtures used by tests. |
| seisfetch/s3.py | Deterministic submission-order joining for concurrent S3 fetches. |
| seisfetch/fdsn.py | Deterministic submission-order joining across multiple FDSN providers. |
| seisfetch/convert.py | Segment-aware parsing, gap/overlap/trim APIs, gap-aware to_dict, xarray merge behavior. |
| seisfetch/contrib/noisepy_adapter.py | Adds ObsPy-free numpy/scipy ports + NoisePy-shaped adapters/store. |
| seisfetch/contrib/init.py | Declares contrib namespace as optional integrations. |
| seisfetch/client.py | Adds trim option (default on) for window-precise parsing results. |
| seisfetch/bulk.py | Ensures deterministic bulk result ordering (submission order) after concurrency. |
| seisfetch/init.py | Introduces PEP 562 lazy imports for transport layers to improve cold-start. |
| pyproject.toml | Pins pymseed range and adds noisepy optional extra (scipy). |
| docs/noisepy-obspy-replacement-report.md | Documents evaluation results, methodology, risks, and recommendation. |
| benchmarks/suites.py | Adds offline benchmark suites and supporting helpers. |
| benchmarks/runner.py | CLI runner to execute suites and persist results JSON. |
| benchmarks/results/m1-native_2026-08-03.json | Captured benchmark results for m1-native run. |
| benchmarks/results/lambda-512m_2026-08-03.json | Captured benchmark results under lambda-512m limits. |
| benchmarks/results/lambda-1g_2026-08-03.json | Captured benchmark results under lambda-1g limits. |
| benchmarks/results/fargate-class_2026-08-03.json | Captured benchmark results under fargate-class limits. |
| benchmarks/results/ccf_equivalence_m1_2026-08-03.json | Captured CCF equivalence metrics JSON. |
| benchmarks/RESULTS.md | Rendered benchmark summary markdown from results JSON. |
| benchmarks/render_results.py | Deterministic renderer for benchmarks/RESULTS.md. |
| benchmarks/noisepy_eval/run_ccf_eval.py | Harness to compare obspy+noisepy vs seisfetch+adapter through real noisepy. |
| benchmarks/docker/run_matrix.sh | Script to run benchmark matrix under cgroup limits via Docker. |
| benchmarks/docker/Dockerfile.bench | Benchmark container (installs both seisfetch and obspy). |
| .github/workflows/test.yml | Adds CI job running offline pytest matrix (ubuntu/macos, py3.10/3.12). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+219
to
+230
| # keep the data dtype when fill_value is exactly representable | ||
| # in it (0 in float32 stays float32 — obspy merge semantics); | ||
| # promote otherwise (NaN into int data -> float64) | ||
| data_dtype = segs[0].data.dtype | ||
| cast = np.asarray(fill_value).astype(data_dtype, casting="unsafe") | ||
| try: | ||
| fits = bool(cast == fill_value) or ( | ||
| np.isnan(fill_value) and np.isnan(cast) | ||
| ) | ||
| except (TypeError, ValueError): | ||
| fits = False | ||
| dtype = data_dtype if fits else np.result_type(data_dtype, np.float64) |
Comment on lines
+283
to
+289
| sr = cur.sampling_rate | ||
| if sr <= 0: | ||
| continue | ||
| sample_interval_ns = int(1e9 / sr) | ||
| expected_next_ns = cur.endtime_ns + sample_interval_ns | ||
| overlap_ns = expected_next_ns - nxt.starttime_ns | ||
| overlap_samples = overlap_ns / sample_interval_ns |
Comment on lines
+13
to
+16
| def _day_bytes(day_marker: int) -> bytes: | ||
| # deterministic distinct content per day via npts | ||
| np.random.seed(day_marker) | ||
| return make_mseed(network="IU", station="ANMO", npts=500 + day_marker) |
seisfetch/contrib/response.py (~450 lines, numpy + stdlib xml): - evalresp-equivalent evaluate_response (analog/Hz/digital PZ, FIR with DC normalization and CorrectionApplied phase, stage gains, (iw)^n unit conversion, CM/MM/NM scaling): max rel diff 1.6e-10 vs compiled evalresp on both CI.PASC epochs, all outputs - discovery, proven by perturbation: evalresp IGNORES the XML NormalizationFactor/Frequency and recomputes A0 at the STAGE GAIN's frequency (2007 epoch: f_norm 0.03 Hz vs gain freq 1.0 Hz reproduces evalresp's 0.997788169 scalar to 9 digits) - remove_response_np: full obspy Trace.remove_response port (SAC taper, _npts2nfft, pre_filt, water level) — 6.6e-16 of peak vs obspy on the real 6.9M-sample Tohoku day, and 1.9 s vs obspy's 3.6 s - translate_resp_np + damped_oscillator_response: SeisIO.jl-style translation (eps-guard, target-response stabilization, PZ-only) - mode='paz' quantified: 0.7-1.3% below 4 Hz, unusable >= 16 Hz - design doc docs/response-removal-design.md; 10 precision tests; 19 KB two-epoch real StationXML fixture Container parse gap (profiled, fixed): - root cause was np_datasamples.copy() of the borrowed C buffer (27-74 ms under cgroups), not per-segment Python; fixed by decoding via create_numpy_array_from_recordlist into a numpy-owned array: 21 ms in EVERY environment (obspy 32-39 ms), parse RSS 92 -> 37 MiB - benchmarks/profile_parse.py micro-harness; docs/pymseed-issue-draft.md (upstream enhancement proposal, NOT submitted); report hypothesis bullet corrected Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Matrix rerun: parse now fastest in every environment (21.4 ms native / ~20 ms all containers vs obspy 32-37 ms). bench suite gains an untimed warmup call per timing loop (first-touch page faults skewed macOS min-of-5 by ~2x). Report tables updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ctra Executed, fully offline (committed fixtures): obspy reference vs seisfetch full/paz/translate on the Tohoku day at CI.PASC. Offset waveform overlay, time-domain residual envelopes (full at float64 eps), amplitude + residual spectra with the pre_filt passband marked. Validated categorical palette; listed in notebooks/README. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- translate_resp_np gains pre_filt (applied in the same spectral pass, same pipeline point as obspy) — with identical demean+SAC taper the translation now agrees with water-level removal to ~1e-7 of the spectrum across the passband (pure stabilization difference); the earlier two-pass band-limit had injected truncation leakage at 1-8 Hz - paz mode renormalized at the sensitivity frequency (|H(f_sens)| == sensitivity by definition): removes stale-A0 scalar bias, leaving only genuine FIR ripple (2011 epoch long-period error 1.3% -> 0.008%) - notebook: log-binned Fourier amplitude spectra in log-log; all four methods now overlay one-to-one; captions updated Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ponse specialist) Deduplicated synthesis of three adversarial reviews with reproduced findings: 4 blockers (LGPL derivation of the obspy ports, silent empty-bytes failure contract, dead location-wildcard defaults on both backends, response-module silent failures on dirty metadata incl. the conditional-A0 correction to our evalresp claim), ranked majors across correctness/operations/sustainability, verified praise, and a recommended fix order. No fixes applied — review only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- ObsPy-derived translations (taper, Fourier resample, _npts2nfft, SAC cosine tapers, water-level invert_spectrum) isolated into seisfetch/contrib/obspy_ports.py under SPDX LGPL-3.0-only with a derivation header — the minimal, explicit LGPL surface; re-exported from response.py / noisepy_adapter.py so the public API is unchanged - pyproject: license = 'MIT AND LGPL-3.0-only' (PEP 639), setuptools floor raised to >=77 for license-expression support; verified License-Expression metadata builds correctly - THIRD_PARTY_NOTICES: new 'Derived code' section — ObsPy (LGPL-3.0), NoisePy preprocessing semantics (MIT, Denolle & Jiang), SeisIO.jl translation approach (MIT); evalresp evaluator documented as clean-room (black-box verification, no source translated) - README license section updated; critique doc resolution log added - all 194 offline tests pass unchanged Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…windows B2 — failure contract (silent data loss eliminated): - new seisfetch/exceptions.py: SeisfetchError, FetchError (per-key failures with error class), NoDataError, FDSNError; lazily exported - S3OpenClient/S3AuthClient: per-key outcomes classified — clean 404s tolerated per key, ANY other failure (403/throttle/credentials/ transport) raises FetchError (on_error='warn' opts out); all-missing raises NoDataError unless missing_ok=True - FDSN: 204/404 (nodata=404 requested) map to b'' as no-data; real HTTP errors raise FDSNError on both httpx and urllib paths (urllib now also honors the timeout); FDSNMultiClient raises FetchError when every provider fails; ObspyFDSNClient only maps FDSNNoDataException to b'', everything else propagates - bulk results carry the error class name, machine-readably - benchmarks bench_s3_pull target fixed (loc '' -> '00'; it was a nonexistent key reported as a successful 0-byte pull) B3 — request semantics: - per-channel archives (SCEDC/NCEDC): location='*' (the default) and channel wildcards now resolved by PAGINATED LIST discovery per station-day — location-coded channels (00/10) are found instead of guessed at, and the 404-guess GET amplification is gone - FDSN: '*' passes through as a true wildcard; '' maps to '--' (blank) - date_range is half-open [start, end): a request ending at midnight no longer fetches the following day's objects (was ~2x GETs/egress on every default one-day request) 15 new tests (moto wildcard discovery, error classification, half-open windows, respx FDSN semantics); 3 date_range tests updated to the new contract; 211 passing total. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- CONDITIONAL A0 (the review's correction to our evalresp claim): mode='full' now uses the XML NormalizationFactor as-is when NormalizationFrequency == StageGain/Frequency and recomputes at the gain frequency only when they differ — both branches verified against compiled evalresp on deliberately corrupted A0 (2x) to <1e-9. paz mode keeps its always-renormalize-at-f_sens rule, now documented as a deliberate divergence. - Epoch selection: timestamps parsed to UTC-aware datetimes; '-08:00' offsets no longer select the wrong epoch; 'Z'/'+00:00'/naive agree; '--' location normalized to blank. - Silent-NaN family eliminated: _ref_magnitude raises when the renormalization reference sits on a zero/pole (e.g. bogus 0.0 Hz gain frequency); zero-sum FIR raises; degenerate short-segment SAC taper now matches obspy bit-exactly instead of producing NaN edges (remove_response_np on a 19-sample stub is finite). - is-None semantics: zero StageGain/sensitivity/A0 raise as broken-metadata sentinels; missing StageGain/Value raises (evalresp rejects it too); missing InstrumentSensitivity raises in paz mode; FIR/digital-PZ stages without Decimation/InputSampleRate raise with the stage number named at parse time. - Polynomial/ResponseList stages raise NotImplementedError instead of silently degrading to a gain-only stage. - DEF output supported (native quantity); 'M/S/S' unit alias accepted. - Docs corrected (design doc + notebook re-executed with the conditional-A0 wording); 14 new dirty-metadata tests; 225 passing. All four critique blockers now closed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ty, guards - to_dict(fill_value=): buffer sized from MAX end time (contained segments crashed it before) and obspy merge(method=1) containment policy implemented — a segment fully contained in already-placed data is skipped (surrounding trace wins), partial tail overlap stays later-overwrites; verified array-equal against obspy on the reviewer's contained and tail-overlap reproductions. merge_fill0_np / bundle_to_xarray / to_zarr inherit the fix. - Mixed sampling rates under one NSLC raise MixedSamplingRateError (typed, names the rates) from to_dict and metadata; segments() is the documented per-rate escape hatch. - Per-record fallback now sorts records by start time before contiguity merging: out-of-order contiguous records heal to the same topology as libmseed's tracelist, so the >100-segment rejection can no longer disagree between paths on identical bytes. - Truncated buffers warn (N trailing bytes not parsed): exact count on the fallback path; 128-byte-alignment heuristic gates a record-list walk on the fast path so the hot path stays fast. - bundle_to_obspy/get_waveforms: merge=None default returns one Trace per segment like obspy.read (filter/detrend work on gappy data); merge=1 restores the old masked force-merge. - preprocess_raw_np: sample-alignment runtime guard — sub-sample-offset windows raise with an instructive message instead of diverging totally from the obspy chain (reviewer measured max abs diff 687.9 on a 0.4-sample offset). 13 regression tests in tests/test_correctness_majors.py; 238 passing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- boto3 config on both S3 clients: adaptive retry mode (5 attempts), connect/read timeouts 10/60 s (unreachable buckets no longer hang for minutes), max_pool_connections sized to the thread fan-out - one shared ThreadPoolExecutor per S3 client (context-manager closable) — per-call executors multiplied by bulk fan-out used to push up to 128 concurrent GETs through a 10-connection pool - list_networks/list_stations paginate; tested past the 1000-key truncation that poisoned bulk-job discovery - routing: BG (The Geysers) corrected SCEDC -> NCEDC; anonymous EarthScope AccessDenied failures now carry a backend='s3_auth' hint - S3AuthClient: 45-minute credential refresh clock + one retry on ExpiredToken/InvalidToken — multi-hour bulk jobs no longer die when short-lived EarthScope credentials expire mid-run - FDSNMultiClient: strategy='failover' default — providers queried IN ORDER, first non-empty wins (verified the second provider is never contacted on success); 'broadcast' (old behavior: 4x load on community services + duplicate records) is an explicit opt-in; all-providers-fail raises FetchError - bulk memory hygiene: fetch_bulk_numpy drops raw bytes after parsing (keep_raw=True restores; nbytes accounting preserved so success/ throughput still work); new iter_bulk_raw generator streams results in completion order for campaign-scale jobs - get_numpy filters to the requested channel/location after parse — EarthScope station-day objects no longer return every channel regardless of the request 12 new tests in tests/test_operations.py; 250 passing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ed results - Version 0.3.0 with a single source (importlib.metadata; unused setuptools-scm dropped from build requires); comprehensive CHANGELOG covering the 0.3.0 breaking/behavior changes - pymseed pin raised to >=0.6,<0.10 after verifying 0.9.3 (45 parse tests + private-API sid fallback + owned-buffer decode all pass); issue draft notes the 0.9.3 verification - CI: lint job (ruff check + format over seisfetch/tests/benchmarks), pytest matrix extended to 3.9/3.10/3.12/3.13 + macOS, and an informational continue-on-error leg that tests the parse path against the LATEST pymseed so the pin never silently strands users - Report traceability restored: cold-import row now shows the committed JSON values (0.08 vs 0.13 s — a lambda-container number had been spliced into the native row) and the footprint claim (80.4 vs 311.4 MB) now has a committed footprint suite JSON behind it - Benchmark results are now READABLE: render_results generates grouped bar plots (SVG, validated palette, direct value labels; parse/cold import/memory/footprint across all four machines with the 250 MB Lambda layer line drawn) embedded in RESULTS.md — GitHub renders MD+SVG natively — plus a self-contained stdlib-generated RESULTS.html twin with inlined images. matplotlib stays a dev extra; zero new core dependencies - Benchmark provenance: containers now record the git sha (SEISFETCH_SHA passed by run_matrix.sh) and the EFFECTIVE cgroup cpu limit instead of the host core count; auxiliary JSONs (CCF equivalence) no longer render as a stale 'unknown' machine section - Hygiene: duplicate 11 MB fixture removed (tests/test_local.mseed was the same git blob as bench.mseed), ~15 diagnostic scripts + 4 PNGs moved from tests/ to tools/diagnostics/, .DS_Store untracked, README links benchmarks/CHANGELOG/notebook and corrects '37+' to '30+ FDSN servers' Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The lint job installed unpinned latest ruff while pre-commit pinned v0.8.6 and pixi shipped 0.15.9 — three formatters disagreeing. CI's newer ruff failed format --check on style-rule changes (operator spacing in f-strings, implicit string-concat collapsing, assert message parenthesization). Now: CI installs ruff==0.15.9, pre-commit rev bumped to v0.15.9 (tag verified), and the repo is reformatted with that exact version (6 files). All pytest legs were already green — lint was the only failure on the PR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…dist - .github/workflows/publish.yml: on v* tags, build + twine check + publish via PyPI Trusted Publishing (OIDC, environment 'pypi'); one-time pypi.org publisher configuration documented in the file - MANIFEST.in: sdist pruned from 10.4 MB (11 MB test fixture, executed notebooks, benchmarks) to 71 KB — source + licenses + CHANGELOG only Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.