From 9fcf3f1b638950c84230bc05798ac834916cd03a Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Wed, 9 Sep 2026 19:59:21 -0400 Subject: [PATCH 1/2] feat(cfis): wire the UNIONS external masks into the workflow PR #847 removed ShapePipe's mask generation and left the sky-fixed UNIONS masks to be queried per object by make_cat via MASK_EXT_PATHS. The committed config never set it, so no campaign wrote a mask column and the merged shear catalogue carried no mask information at all. It does now. The DR6 Aug-2026 ugriz bit ladder is staged group-readably at /project/def-mjhudson/unions-wl/masks/dr6-2026-08 (11 boolean healsparse maps, nside 131072, True = masked), reached as a third input root SP_INPUT_MASKS beside tiles and exposures. config_tile_Mc.ini names all 11 in MASK_EXT_PATHS and carries the bit table; final_cat.param carries the matching MASK_n1 .. MASK_n2048, replacing the IMAFLAGS_ISO note. Two caveats live with the config, because a cut written without them is wrong: the August regeneration swapped the content of bits 0 and 1, so n1|n2 is only safe combined; and n2048 is 1 where there is no Pan-STARRS z2 data, so an OR over every column masks the whole survey. Nothing cuts on these here. PSF-star selection is untouched: MASK_PATHS in config_exp_psfex.ini stays commented out. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar --- workflow/README.md | 10 +++++++ workflow/Snakefile | 7 ++--- workflow/config.yaml | 7 ++++- workflow/config/cfis/config_tile_Mc.ini | 36 +++++++++++++++++++++++++ workflow/config/cfis/final_cat.param | 34 ++++++++++++++--------- 5 files changed, 77 insertions(+), 17 deletions(-) diff --git a/workflow/README.md b/workflow/README.md index c00d66c18..6b21ef097 100644 --- a/workflow/README.md +++ b/workflow/README.md @@ -221,6 +221,16 @@ profiles/nibi/config.yaml SLURM executor; apptainer SDM; per-user jobs cap; kee catalogue server, staged, or rasterized, which is why the old `star_catalogue` / `exp_star_cat` / `exp_mask` rules and their cache root are gone. +- **External masks are wired, on the tile side only.** `inputs.masks` is a third + input root beside tiles and exposures, exported as `$SP_INPUT_MASKS` and + pointing at the UNIONS DR6 ugriz bit ladder: one boolean healsparse map per + bit, nside 131072, `True` = masked. `config_tile_Mc.ini` names all 11 of them + in `MASK_EXT_PATHS`, so `make_cat` writes `MASK_n1` … `MASK_n2048` and + `final_cat.param` carries the matching 11 names. That file and the config + hold the bit table and the two caveats (the August bit-0/1 halo swap is + unconfirmed, so use `n1|n2` combined; `n2048` is 1 where there is *no* + Pan-STARRS z2 data, so an OR over every column masks everything). Nothing + cuts on them here. - **The index is parse-time data, never a rule input.** Appending tiles changes which jobs exist without invalidating completed work. - **Exposure products are not `temp()`.** Exposures overlap tiles, so diff --git a/workflow/Snakefile b/workflow/Snakefile index 4317e871f..64640f615 100644 --- a/workflow/Snakefile +++ b/workflow/Snakefile @@ -1000,9 +1000,9 @@ def unit_pre(stage, unit, *, exp_name=None, forest=None, env=None, There is no per-unit ``cfis`` symlink any more: $SP_CONFIG points straight at the committed config dir. - It also exports the configured input roots as ``SP_INPUT_TILES`` and - ``SP_INPUT_EXPOSURES`` and the PSF choice as ``SP_PSF`` for the committed - ini chain. + It also exports the configured input roots as ``SP_INPUT_TILES``, + ``SP_INPUT_EXPOSURES`` and ``SP_INPUT_MASKS`` and the PSF choice as + ``SP_PSF`` for the committed ini chain. Finally it ``rm -rf``s this stage's own fixed run dir — ShapePipe's FileHandler raises on an existing run dir, and it is how a rerun never sees @@ -1021,6 +1021,7 @@ def unit_pre(stage, unit, *, exp_name=None, forest=None, env=None, f"export SP_CONFIG='{CONFIG_DIR}'", f"export SP_INPUT_TILES='{INPUTS['tiles']}'", f"export SP_INPUT_EXPOSURES='{INPUTS['exposures']}'", + f"export SP_INPUT_MASKS='{INPUTS['masks']}'", f"export SP_PSF='{PSF_MODEL}'", # Also set via apptainer-args in the profile; kept here so a hand-run of # this same line outside snakemake behaves identically. diff --git a/workflow/config.yaml b/workflow/config.yaml index 38979b2d0..f119139be 100644 --- a/workflow/config.yaml +++ b/workflow/config.yaml @@ -19,11 +19,16 @@ tile_list: /project/def-mjhudson/cdaley/sp-products/smk-g6/tiles.txt # Inputs are the only site-specific data paths; all committed configs consume these -# through SP_INPUT_TILES and SP_INPUT_EXPOSURES. +# through SP_INPUT_TILES, SP_INPUT_EXPOSURES and SP_INPUT_MASKS. inputs: # Pre-staged P3 data (get_images RETRIEVE=symlink). tiles: /project/def-mjhudson/unions-wl/tiles exposures: /project/def-mjhudson/unions-wl/exposures + # UNIONS sky-fixed masks: one boolean healsparse map per bit of the + # band-combined ugriz mask (nside 131072, True = masked), queried per object + # by make_cat -- never rasterized. Consumed by MASK_EXT_PATHS in + # config_tile_Mc.ini, which carries the bit table. + masks: /project/def-mjhudson/unions-wl/masks/dr6-2026-08 # The container every job runs inside (apptainer software-deployment in the profile). container: /project/def-mjhudson/cdaley/containers/shapepipe-develop-runtime.sif diff --git a/workflow/config/cfis/config_tile_Mc.ini b/workflow/config/cfis/config_tile_Mc.ini index 5920d0a91..a8316f8b7 100644 --- a/workflow/config/cfis/config_tile_Mc.ini +++ b/workflow/config/cfis/config_tile_Mc.ini @@ -82,3 +82,39 @@ NUMBERING_SCHEME = -000-000 SM_DO_CLASSIFICATION = False SHAPE_MEASUREMENT_TYPE = ngmix + +# The UNIONS sky-fixed masks, queried once per object and written as one +# MASK_