Wire the UNIONS external healsparse masks into the workflow (MASK_n* columns) - #886
Open
cailmdaley wants to merge 2 commits into
Open
Wire the UNIONS external healsparse masks into the workflow (MASK_n* columns)#886cailmdaley wants to merge 2 commits into
cailmdaley wants to merge 2 commits into
Conversation
PR #847 removed ShapePipe's mask generation and left the sky-fixed UNIONS masks to be queried per object by make_cat via MASK_EXT_PATHS. The committed config never set it, so no campaign wrote a mask column and the merged shear catalogue carried no mask information at all. It does now. The DR6 Aug-2026 ugriz bit ladder is staged group-readably at /project/def-mjhudson/unions-wl/masks/dr6-2026-08 (11 boolean healsparse maps, nside 131072, True = masked), reached as a third input root SP_INPUT_MASKS beside tiles and exposures. config_tile_Mc.ini names all 11 in MASK_EXT_PATHS and carries the bit table; final_cat.param carries the matching MASK_n1 .. MASK_n2048, replacing the IMAFLAGS_ISO note. Two caveats live with the config, because a cut written without them is wrong: the August regeneration swapped the content of bits 0 and 1, so n1|n2 is only safe combined; and n2048 is 1 where there is no Pan-STARRS z2 data, so an OR over every column masks the whole survey. Nothing cuts on these here. PSF-star selection is untouched: MASK_PATHS in config_exp_psfex.ini stays commented out. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar
This was referenced Sep 10, 2026
…n type
FITSCatalogue._get_fits_col_type checked `type(col_data[0]) is bool`, which
only matches Python's builtin bool. A numpy bool array's first element is
np.bool_, so it fell through to the float branch and every boolean column
was silently written as float64 {0.0, 1.0}. This hit make_cat's MASK_n*
columns, queried from boolean healsparse maps via mask_query.query_map,
even though the maps, the docs (config_tile_Mc.ini, final_cat.param,
workflow/README.md), and query_map's own docstring all promise a boolean
value (including the off-map sentinel False). Recognizing np.bool_ writes
the FITS 'L' (logical) format instead, matching the promised dtype; no
other code path depended on the float64 widening.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #879. Closes #844's mask half; the rest of #844 (tile-side retention) stays open.
#847 removed ShapePipe's mask generation and left the sky-fixed UNIONS masks to be queried per object by
make_catthroughMASK_EXT_PATHS. The committed config never set it, so no campaign has written a mask column and the merged shear catalogue carries no mask information at all.This wires it. The DR6 Aug-2026 ugriz bit ladder is staged group-readably at
unions-wl/masks/dr6-2026-08— 11 boolean healsparse maps, nside 131072,True= masked — reached as a third input rootSP_INPUT_MASKSbeside tiles and exposures.config_tile_Mc.ininames all 11 and carries the bit table;final_cat.paramcarries the matchingMASK_n1…MASK_n2048.Two caveats live with the config, because a cut written without them is wrong. Whether the August regeneration swapped the content of bits 0 and 1 is unconfirmed, so
n1|n2is only safe combined. Andn2048is 1 where there is no Pan-STARRS z2 data, so an OR over every column masks the whole survey. Nothing cuts on these here; sp_validation'sIMAFLAGS_ISOcut has to move onto them (CosmoStat/sp_validation issue to follow).PSF-star selection is untouched:
MASK_PATHSinconfig_exp_psfex.inistays commented out, keeping the star diet narrow.Verified: all 11 paths expand and exist through the pipeline config reader; the DAG is unchanged at 1090 jobs; querying the staged maps at 1000 positions from an smk-g6 tile returns sane per-bit fractions (n4 star 8.2%, n1 halo 2.6%, n1024 MaxiMask 3.5%, n2048 = 1.0 exactly as expected off the PS z2 footprint).
🤖 Generated with Claude Code
https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar