Read ShapePipe v2 campaign products: hdf5 catalogues, MASK_n* cut, patches retired - #343
Open
cailmdaley wants to merge 17 commits into
Open
Read ShapePipe v2 campaign products: hdf5 catalogues, MASK_n* cut, patches retired#343cailmdaley wants to merge 17 commits into
cailmdaley wants to merge 17 commits into
Conversation
Replace read_hdf5_file's hardcoded patches/<name>/<tile-ID> lookup with find_dataset_group(), which descends from the file root through single container groups until it reaches the per-unit datasets. This reads the legacy patches/<campaign>/ layout that ShapePipe still writes as a compatibility shim, a future flat tiles/ layout, and the exposures/<exp> layout of full_starcat_<campaign>.hdf5 with the same code. read_star_catalogue() keeps the FITS path for files ending in .fits. Requested columns missing from the data now raise a clear KeyError. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar
ShapePipe v2 drops IMAFLAGS_ISO for eleven boolean MASK_n* columns. galaxy.mask_cut() ORs a configurable list of them (default MASK_n4, MASK_n1, MASK_n2, MASK_n8, MASK_n1024 — stars, star halos, manual galaxy mask, MaxiMask) and returns the keep mask; a catalogue missing any of the requested columns raises a KeyError naming them. classification_galaxy_base takes mask_columns; extract_info.py passes the params.py mask_columns list and uses it for the star-sample cut too, and now reads both catalogues through the new readers. Column lists in params.py and masking.py's SPATIAL_CUTS updated to the MASK_n* names. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar
ShapePipe v2 processes a campaign (a tile list); P1-P7 no longer exist.
- catalog_builders.JointCat: get_patches()/get_n_obj() and the per-patch
FITS merge are replaced by a merge over a list of campaign hdf5 files
(-i final_cat_A.hdf5+final_cat_B.hdf5), read through
read_campaign_catalogue. The 'patch' int8 column becomes a 'campaign'
string column; the hdf5 root attr becomes 'campaigns'.
- survey.get_footprint() deleted: it was a lookup table of P1-P7 (plus W3)
RA/Dec boundaries, meaningless for a campaign. Its only caller,
catalog.check_matching, used it as an optional pre-filter via a 'name'
argument that every caller passed as None; the argument goes too, along
with the test_survey test that exercised P5.
- merge_psf_cat.py, combine_results.py, stats_tile_id_gal_counts.py,
compute_area.py: patch vocabulary and v1/v1.5/v1.6 P-name shortcuts
generalised to an explicit list of campaigns.
- params.py: 'name = "P7"' becomes 'campaign = None'.
Deleted (only ever meaningful for the P1-P7 era):
- scripts/prepare_patch_for_spval.sh: symlinks a v1 per-patch run tree
(~/psfex/${patch}/output/run_sp_Ms/.../full_starcat-0000000.fits,
tiles_${patch}.txt) into a working dir; neither the layout nor the file
names exist in v2.
- scripts/plot_rho_stats_patches.py: globs P* directories and reads
P*/output/run_sp_Pl/mccd_plots_runner/output/rho_stats_id.fits, one
curve per patch. No campaign analogue.
- scripts/survey_stats_all.sh: hardcoded 'for patch in P1 ... P7' over v1
run-directory bookkeeping.
- scripts/star_match_stats.py: sums stats_file.txt over the seven patches.
- scripts/check_tile_IDs_SP_LF.py: compares per-patch ShapePipe tile IDs
against CFIS3500_THELI_P<n>.list Lensfit files; both sides P-named.
The word 'patch' survives only in catalog.py's comment naming the legacy
hdf5 group, and in the treecorr jackknife sense (npatch/patch_number).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar
Seventeen unit tests on tiny synthetic hdf5 fixtures built in a temp dir: both campaign layouts (legacy patches/<campaign>/<tile-ID> and flat tiles/<tile-ID>) read identically, param-list restriction, missing-column and ambiguous-layout errors; the star reader on exposures/<exp> hdf5 and on FITS; galaxy.mask_cut defaults, explicit list, empty list, missing column, and a v1 IMAFLAGS_ISO-only catalogue; JointCat.merge_catalogues across two campaigns of different layouts, plus its error paths. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar
- concatenate_datasets preallocates the output and fills it column by column, so peak memory is the packed output plus one tile instead of the full-width uncut catalogue; the verbose estimate uses the real itemsize. - validate the requested columns against every tile dataset, not only the first, and name the offending dataset in the error. - read_campaign_catalogue checks the root n_tiles attribute and refuses a truncated file; campaign_shape reports row count and dtype from metadata. - JointCat.merge_catalogues preallocates the merged array from that first pass (no more accumulate-then-concatenate, which doubled peak memory), promotes each column's dtype across all campaigns so a wider string or integer column in a later file is no longer silently truncated, and refuses multi-dimensional columns explicitly. - reduce_mem reduces int32 to int16 (int8 wrapped N_EPOCH/CCD_NB values) and every assignment is range-checked, raising instead of wrapping. - check_matching drops the identity index over d1 that the retired footprint prefilter left behind, and extract_info applies mask_cut to the matched subset instead of the whole catalogue. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar
Add config/calibration/mask_v2.0.yaml: the v1.X.11 cut set with IMAFLAGS_ISO and the v1 post-processing masks replaced by the boolean MASK_n<bit> columns (True = masked, hence kind: equal, value: False), the coverage bits and MASK_n2048 listed but commented out. The v1.X configs are left untouched: each describes a legacy catalogue that really has IMAFLAGS_ISO, and rewriting them would break reproducing published versions. plots.sky_plots no longer hardcodes the v1 label set; it combines whichever of IMAFLAGS_ISO / MASK_n* / npoint3 / 1024_Maximask the config declared. The comprehensive-to-minimal demo asks for the v2 mask labels. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar
- params.py interpolated the removed 'name' into two paths (NameError on import); use 'campaign', give it a real default, and point star_cat_path at full_starcat_<campaign>.hdf5 (hdu_star_cat now documented as legacy FITS only). params_im_sim.py renames 'name' to 'campaign' likewise. - combine_results.get_area matches both the campaign and the legacy patch wording, and raises on a missing file or unmatched pattern instead of returning None / a 1 deg^2 placeholder that silently rescales densities. - merge_psf_cat writes the campaign *name* as a string column (FITS 'A<n>'), matching JointCat; the old 1-based ordinal depended on -p argument order. - compute_m_bias_image_sims counts tiles via the n_tiles attribute or find_dataset_group, not a hardcoded legacy group inside a bare except. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar
Tests: assert the exact concatenation instead of sorted values, add a fixture whose keys are inserted out of order, and cover the truncated n_tiles file, a column missing from a later tile, dtype promotion across campaigns in both argument orders, reduce_mem overflow, and a multi-dimensional column. Docs: CLAUDE.md, scripts/calibration/README.md, homogenize_cat_extended.py and the catalog_builders docstrings now say campaign. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar
Three defects in the campaign reading path, found in review: - concatenate_datasets and campaign_shape both took the output dtype from the first dataset alone, so a campaign whose tiles differ (S7 next to S12 tile IDs, i2 next to i4, f4 next to f8 after a partial reprocessing) had the later tiles silently truncated, downcast or wrapped. np.concatenate, which this code replaced, promoted. Both now build the dtype with group_dtype(), promoting every column across every dataset; catalog_builders._promote becomes an alias of the shared catalog.promote_dtypes rather than a second copy of it. - merge_catalogues still held a whole campaign in memory next to the preallocated output, the very thing its comment claimed the rewrite avoided -- and with one campaign per merge in v2, that is the normal case, ~2x the merged catalogue at DR6 scale. It now fills the output tile by tile through the new catalog.iter_campaign_tiles(), so peak memory is the output plus a single tile. - write_hdf5_file wrote the merged array twice, create_dataset(data=...) followed by an immediate dset[:] = dat_all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar
mask_v2.0.yaml dropped v1.X's '64_r' r-band imaging cut without a replacement, so a v2 calibration run admitted objects outside the r-band footprint that v1 rejected -- a silent change of effective area, n(z) and galaxy-density normalisation. Enable MASK_n64, v1's '64_r' equivalent. v1's other coverage cut, npoint3 >= 3, came from an external post-processing catalogue and has no v2 counterpart; say so in the config rather than leaving its absence unexplained. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar
extract_info fancy-indexed the full-width catalogue, dd[ind_star], only to read ~5 boolean mask columns from it -- a copy of every column for every matched star (~GB at DR6 scale). Mask first, index the resulting bool array. plot_leakage still labelled its curves "all", "P1" ... "P7": a P-named survivor of the #340 retirement, and a fixed length that silently mismatched the number of input files. Labels and colours now follow the input files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar
The v2 branch inferred from v1's '64_r' naming that MASK_n64 flags r-band
imaging coverage. It does not: bit 64 is an undocumented *reason* bit of
the r-band default bitmask, and OR{n1,n2,n4,n8,n64,n1024} reproduces
mask_r, the v1 r-band mask, exactly on the P3 region.
Add MASK_n64 to DEFAULT_MASK_COLUMNS so the default galaxy cut is exactly
that set, documented as reproducing mask_r, and mirror it in the
calibration params, the v2 mask config and the minimal-catalogue demo.
Document n16/n32/n128/n256 as the u/g/i/z coverage flags (no r flag: the
catalogue is r-selected) and n2048 as absent Pan-STARRS z2. The faint vs
bright assignment of n1/n2 is unconfirmed for the Aug-2026 products, so
the labels no longer claim one.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar
Pushed by the lint gate.
Contributor
🔴 ruff found lint / format issues@cailmdaley — these block the merge into
|
ShapePipe's make_cat pre-fills the NGMIX_* columns with sentinels (G1/G2 = -10, T/FLUX = 0) and overwrites them only for objects present in the ngmix output, so an object ngmix never fit keeps NGMIX_MCAL_FLAGS == 0 and passes a flag-only cut. In final_cat_smk-g7.hdf5 that is 18,983 of 1,851,100 objects (1.03%); admitting them drags mean e1 to -0.096 (std 0.98) from +0.0001. classification_galaxy_ngmix already rejected all 18,983 via the NGMIX_G1_PSF_ORIG_NOSHEAR != -10 guard, so the production selection was never affected -- verified on the real file, which gives 1,105,851 rows out with and without the new cut. But that protection was incidental: it is an exact float equality against a sentinel ShapePipe may change, and the coadd N_EPOCH >= 2 cut in classification_galaxy_base does not substitute for it (18,750 of the 18,983 have N_EPOCH >= 1). Cut on NGMIX_N_EPOCH > 0 explicitly so the guarantee is stated, not inferred. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar
mask_cut decided on truthiness via astype(bool). ShapePipe writes the
MASK_n* columns as float64 {0, 1} rather than bool (being fixed
upstream), and astype(bool) reads NaN as True, so an incomplete mask
column would have silently deleted sky. Decide on the value instead
(masked iff > 0.5), accept bool, int and float alike, and treat NaN as
"no verdict recorded" -- keep the object, but count and warn, since a
nonzero count means the product is defective. final_cat_smk-g7.hdf5
carries no NaNs and only exact 0.0/1.0, so this is defensive: the real
file gives 1,105,851 rows out before and after.
read_star_catalogue silently accepted a truncated file and threw away
exposure provenance. It now validates the n_exposures root attribute
against the datasets found, as the galaxy reader validates n_tiles
(check_n_tiles generalised to check_n_units), and adds an EXPID column
carrying the exposure number each star came from. The datasets are
named by that number and concatenating them discarded it, leaving no
way to group stars by exposure downstream. Names may be bare
("2086324", as smk-g7 writes them) or carry the CFIS suffix
("2110000p"), so EXPID takes the leading digits. On the real star
catalogue: 53,264 stars over 127 exposures, matching n_exposures.
Also note in group_dtype that ShapePipe writes TILE_ID as f8, so the
string-promotion branch is for a future string-valued TILE_ID, with a
TODO recording that as an open schema decision. No behaviour change.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar
Pushed by the lint gate.
Collaborator
Author
|
Exercised against the first real v2 products (smk-g7, 64 tiles): galaxy reader 1,851,100 rows / 77 columns in 11 s at 1.2 GB RSS; star reader 53,264 stars over 127 exposures; default mask cut keeps 72.98%; full selection 1,105,851 rows with mean e1 = −0.00020 ± 0.00019, e2 = −0.00015 ± 0.00019. Three follow-up commits from that run: an explicit — Claude (Fable) on behalf of Cail |
mask_v2.0.yaml, applied downstream to the comprehensive catalogue, cut on neither NGMIX_MCAL_FLAGS nor any epoch column. It rejected the never-fit objects only through its NGMIX_G1/G2_PSF_ORIG_NOSHEAR != -10 cuts, and only because make_cat happens to fill the PSF columns from the same -10 literal it uses for the galaxy ellipticities. That is the same accidental immunity just removed from classification_galaxy_ngmix, one stage further downstream. Add NGMIX_N_EPOCH >= 1. The column is already carried into the comprehensive catalogue via add_cols_pre_cal in params.py. On final_cat_smk-g7.hdf5 the cut keeps 1,832,117 of 1,851,100 objects, removing exactly the 18,983 (1.03%) never-fit rows. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adapts sp_validation to the ShapePipe v2 campaign products (CosmoStat/shapepipe#879, #886). Catalogues now arrive as
final_cat_<campaign>.hdf5andfull_starcat_<campaign>.hdf5, patches P1–P7 are retired (#340), and the singleIMAFLAGS_ISObitmask is replaced by per-reason booleanMASK_n*columns (#342).The readers are layout-agnostic:
read_campaign_catalogueaccepts both the legacypatches/<campaign>/<tile-ID>and a flattiles/<tile-ID>group, validates the requested columns against every tile rather than only the first, and refuses a file whose contents disagree with itsn_tilesattribute. Reading and merging preallocate the output and fill it tile by tile, so peak memory is the merged array plus one tile, and each column's dtype is promoted across every tile and campaign instead of inherited from the first. The galaxy cut is config-driven:galaxy.mask_cutORs a configurable list of mask columns, defaulting to n1, n2, n4, n8, n64, n1024 — the reason bits of the r-band default bitmask, whose OR reproduces v1'smask_rexactly on the P3 region.config/calibration/mask_v2.0.yamlmirrors that set.Deleted: five scripts and
survey.get_footprint, all of which only ever addressed the P1–P7 run-tree layout and file names, neither of which exist in v2.Deliberately left alone: the v1.x mask configs, which describe legacy catalogues that really do carry
IMAFLAGS_ISOand are needed to reproduce the published catalogue versions.Tests: 237 passed, 3 failed, 9 skipped. The three failures are pre-existing and unrelated — one CAMB reference drift and two from the missing pymaster;
test_pseudo_cl.pyandtest_cosmo_val.pycould not be collected for the same reason. The readers are verified on synthetic files only so far; the first real v2 products come from the smk-g7 campaign now running on Nibi, and this PR should be exercised against them before merging.— Claude (Fable) on behalf of Cail
🤖 Generated with Claude Code
https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar