Snakemake orchestration for the real-data pipeline

.github/workflows/deploy-image.yml

@@ -83,12 +83,11 @@ jobs:
 
       # Smoke-test the binaries baked into the runtime image. Catches the
       # class of regression where the image builds but a runtime tool
-      # (sextractor, weightwatcher) is missing or unrunnable.
+      # (sextractor, psfex) is missing or unrunnable.
       - name: Test runtime — binaries
         run: |
           IMAGE=$(echo "${{ steps.meta-runtime.outputs.tags }}" | head -n1)
           docker run --rm "$IMAGE" source-extractor --version
-          docker run --rm "$IMAGE" weightwatcher --version
           docker run --rm "$IMAGE" psfex --version
 
       - name: Test runtime — shapepipe entry point (read-only fs)

.gitignore

@@ -139,3 +139,4 @@ code
 # .felt here is a machine-local symlink into it. Never track it in this repo.
 /.felt/
 /.felt
+.snakemake/

Dockerfile

@@ -34,7 +34,7 @@
     COVERAGE_FILE=/tmp/.coverage
 
 # System dependencies — three categories:
-#  - astromatic binaries (psfex, source-extractor, weightwatcher) ship as
+#  - astromatic binaries (psfex, source-extractor) ship as
 #    Debian packages on bookworm; preferred over building from source.
 #  - compilers and dev libs needed to build the heavier wheels (galsim,
 #    mpi4py, python-pysap, fitsio).
@@ -55,7 +55,7 @@
         libcfitsio-dev \
         libproj-dev proj-bin \
         libgl1-mesa-glx \
-        psfex source-extractor weightwatcher && \
+        psfex source-extractor && \
     apt-get clean && rm -rf /var/lib/apt/lists/*
 
 # OpenMPI from source — required for hybrid Apptainer MPI on HPC clusters.
@@ -89,7 +89,7 @@
         --disable-dlopen --disable-sphinx && \
     make -j"$(nproc)" && make install && \
     cd / && rm -rf /tmp/openmpi-*
 ENV PATH="/opt/ompi/bin:${PATH}" \
     LD_LIBRARY_PATH="/opt/ompi/lib:${LD_LIBRARY_PATH}"
 
 # uv — fast reproducible Python deps installer. pyproject.toml + uv.lock

docs/source/container.md

@@ -165,7 +165,7 @@ The Dockerfile does **not** duplicate Python deps — those come from
 The asymmetry is deliberate: Python deps go through pyproject + lockfile
 (reproducible, auditable), system deps go through Dockerfile (Debian's
 versioning). Don't `apt install` something that has a Python wheel; don't
-`pip install` something Debian packages directly (e.g. `weightwatcher`).
+`pip install` something Debian packages directly (e.g. `source-extractor`).
 
 ## Why this shape
 
@@ -175,8 +175,8 @@ versioning). Don't `apt install` something that has a Python wheel; don't
 - **`uv sync --frozen`** at build time means the image is bit-exactly
   reproducible from a tagged commit, and impossible to ship with a stale
   lockfile.
-- **Astromatic binaries from Debian** (`psfex`, `source-extractor`,
-  `weightwatcher`) instead of source builds — Debian carries the
+- **Astromatic binaries from Debian** (`psfex`, `source-extractor`)
+  instead of source builds — Debian carries the
   GCC-compatibility patches that the previous Dockerfile had to apply
   inline with `sed`.
 - **Two targets** so canfar batch deployments stay slim while interactive

docs/source/dependencies.md

@@ -57,7 +57,6 @@ packages (no source builds), plus the MPI stack:
 |---------|------------|
 | [Source Extractor](https://www.astromatic.net/software/sextractor/) | {cite:p}`bertin:96` |
 | [PSFEx](https://www.astromatic.net/software/psfex/) | {cite:p}`bertin:11` |
-| [WeightWatcher](https://www.astromatic.net/software/weightwatcher/) | {cite:p}`marmo:08` |
 | OpenMPI (5.0.x) | |
 
 Python dependencies themselves are managed with [uv](https://docs.astral.sh/uv/);

docs/source/installation.md

@@ -38,8 +38,8 @@ docker pull ghcr.io/cosmostat/shapepipe:develop-runtime
 We do not currently build images for Apple Silicon/amr64; however the amd64 images should work on these systems, albeit with reduced performance.
 ```
 
-The image bundles the astromatic binaries (`source-extractor`, `psfex`,
-`weightwatcher`), MPI (`mpi4py` + OpenMPI), and every Python dependency, so
+The image bundles the astromatic binaries (`source-extractor`, `psfex`),
+MPI (`mpi4py` + OpenMPI), and every Python dependency, so
 there is nothing else to install or build. To process data on a cluster with
 MPI, run the pipeline through Apptainer the same way you would any MPI job.
 

docs/source/pipeline_canfar.md

@@ -170,22 +170,16 @@ The downloaded tile weights are compressed. The following call uncompresses all.
 shapepipe_run -c cfis/config_tile_Uz.ini
 ```
 
-### Mask tiles
+### Masks
 
-This step is done globally for all tiles. There might be job failures or interruptions. The following
-command to the `ShapePipe` job script can be run repeatedly; already created masks will be skipped.
-
-```bash
-job_sp_canfar.bash -p $psf -n $OMP_NUM_THREADS -j 4
-```
-
-If masks were created in more than one run, i.e. situated in more than one output directory, these have to be
-combined for subsequent pipeline module runs. This is done by creating a new output directory with symbolic
-links, using the script
-
-```bash
-combine_runs.bash -c flag_tile
-```
+There is no masking step. `ShapePipe` generates no masks: the sky-fixed
+healsparse maps are queried once per object, by `mask_query` on the exposure
+catalogues (`MASK_EXT`, recorded but not cut on) and by `make_cat` on the tile
+catalogue (`MASK_<band>` columns). Point the `MASK_PATHS` / `MASK_EXT_PATHS`
+config entries at the maps and nothing else is needed — no star-catalogue
+download, no rasterization, no `combine_runs.bash -c flag_*`. The only mask that
+touches pixels is the instrument flag image shipped with each exposure, which
+`split_exp` splits per CCD.
 
 ## Tile detection
 
@@ -205,7 +199,7 @@ canfar_submit_job -j 16 -f tile_numbers.txt -P N_PAR -v -J JMAX
 
 ### Exposure Processing
 
-#### Option 0: Global split and exp masks (deprecated; used for earlier v1.x patch runs)
+#### Option 0: Global split (deprecated; used for earlier v1.x patch runs)
 
 For this option, set `sp_local=0`.
 
@@ -217,21 +211,7 @@ For `sp_local=-` both `mh_local` (0, 1) are ok:
 export mh_local=0
 ```
 
-#### Option 0: Mask exposures (deprecated)
-
-Run repeatedly if necessary:
-
-```bash
-job_sp_canfar.bash -p $psf -n $OMP_NUM_THREADS -j 8
-```
-
-Combine all runs:
-
-```bash
-combine_runs.bash -c flag_exp
-```
-
-### Option 1: Local split and mask exposures (recommended)
+### Option 1: Local split exposures (recommended)
 
 Optional: Enable flags for local split processing and merge header runs as
 
@@ -258,12 +238,6 @@ First, determine the number of maximum jobs with the option `-s` (see above). Th
 canfar_submit_job -j 2 -v -f exp_shdu.txt -v -P N_PAR -J JMAX
 ```
 
-### Mask exposures
-
-```bash
-canfar_submit_job -j 8 -f exp_shdu.txt -v -P N_PAR -J JMAX
-```
-
 ### Exposure detection
 
 ```bash
@@ -400,7 +374,7 @@ Rename to general PSF and star catalogue used for all ("a") sub-versions:
 
 
 ```bash
-cp output/run_sp_Ms/merge_starcat_runner/output/full_starcat-0000000.fits \
+cp output/run_sp_tile_Ms/merge_starcat_runner/output/full_starcat-0000000.fits \
     unions_shapepipe_psf_2024_v1.6.a.fits 
 ```
 

docs/source/pipeline_tutorial.md

@@ -44,11 +44,11 @@ Naming and numbering of the input files can closely follow the original image na
   A stacked image is also called *tile*. These files are used on input by `ShapePipe`.
   The pixel data can contain the observed image, a weight map, or a flag map. Tile images and weights are created in the
   case of CFIS by Stephen Gwyn using a combination of `swarp` and his own software. Examples of file names are
-  `CFIS.316.246.r.fits`, `CFIS.205.267.r.weight.fits.fz`, the latter is a compressed FITS file, see below. Tile flag files
-  are created the mask module of `ShapePipe` (see [Mask images](#mask-images)). The tile ID needs to be modified such that the `.` between the two tile numbers (RA and DEC indicator) is not mistaken for a file extension delimiter. For the same reason, the extension `.fits.fz` is changed to `.fitzfz`. In addition, for
+  `CFIS.316.246.r.fits`, `CFIS.205.267.r.weight.fits.fz`, the latter is a compressed FITS file, see below. Tiles have no flag file
+  (see [Masks](#masks)). The tile ID needs to be modified such that the `.` between the two tile numbers (RA and DEC indicator) is not mistaken for a file extension delimiter. For the same reason, the extension `.fits.fz` is changed to `.fitzfz`. In addition, for
   clarity, we include the string `image` for a tile image type.  
   Default convention: **<image_type>-<tile_number>.fits**  
-  Examples: `CFIS_image-277-282.fits`, `CFIS_weight-274-282.fitsfz`, `pipeline_flag-239-293.fits`
+  Examples: `CFIS_image-277-282.fits`, `CFIS_weight-274-282.fitsfz`
 
 - Database catalogue files  
   For very large files that combine information from multiple tiles or single exposures, `ShapePipe` creates `sqlite`
@@ -128,8 +128,6 @@ for all options.
 This script creates the subdirectory `$SP_RUN/output` to store all pipeline outputs
 (log files, diagnostics, statistics, output images, catalogues, single-exposure headers with WCS information).
 
-Optionally, the subdir `output_star_cat` is created by the used to store the external star catalogues for masking. This is only necessary if the pipeline is run on a cluster without internet connection to access star catalogues. In that case, the star catalogues need to be retrieved outside the pipeline, for example on a login node, and copied to `output_star_cat`.
-
 The job script automaticall performs a number of subsequent calls to the `ShapePipe` executable `shapepipe_run`, as
 ```bash
 shapepipe_run -c $SP_CONFIG/<config>.ini
@@ -189,32 +187,49 @@ Finally, the headers of all single-exposure single-CCD files are merged into a s
 Two output directories are created, `run_sp_Uz` for `uncompress_fits`, and `run_sp_exp_SpMh` for the output of the modules
 `split_exp` (`Sp`) and `merge_headers` (`Mh`).
 
-## Mask images
-
-Run
-```bash
-job_sp TILE_ID -j 4
-```
-to mask tile and single-exposure single-CCD images. Both tasks are performed by two calls to the `mask` runner.
-
-Note that internet access is required for this step, since a reference star catalogue is downloaded.
-
-The output of both masking runs are stored in the output directory `run_sp_MaMa`, with run 1 (2) of
-`mask` corresponding to tiles (exposures).
-
-**Diagnostics:** Open a single-exposure single-CCD image and the corresponding pipeline flag
-in `ds9`, and display both frames next to each other. Example
-```bash
-ds9 image-2113737-10.fits pipeline_flag-2113737-10.fits
-```
-Choose `zoom fit` for both frames, click `scale zscale` for the image, and `color aips0` for the flag, to display something like this:
-
-<img width="250" src="img/diag_mask.png">
-
-By eye the correspondence between the different flag types and the image can be
-seen. Note that the two frames might not match perfectly, since (a) WCS
-information is not available in the flag file FITS headers; (b) the image can
-have a zero-padded pixel border, which is not accounted for by `ds9`.
+## Masks
+
+`ShapePipe` does not generate masks. Sky-fixed masks — star halos, stars,
+manual masks for large galaxies, per-band coverage, MaxiMask defects — are
+supplied as [healsparse](https://healsparse.readthedocs.io) maps and are
+consumed by *querying them at object positions*, never by rasterizing them onto
+pixels. Two modules do the querying, from the same shared lookup
+(`shapepipe.utilities.mask_query`): `mask_query` runs between `sextractor` and
+`setools` on the single-exposure single-CCD catalogues and writes one integer
+`MASK_EXT` column (0 = clean), recording the star-body map (bit 2) against every
+detection; `make_cat` writes one
+`MASK_<band>` column per band onto the final tile catalogue, carrying the map
+value verbatim so downstream selections choose their own cuts. Map paths and
+bit selections live in the config files (`MASK_PATHS` / `MASK_BITS` and
+`MASK_EXT_PATHS`), so regenerated mask products cost a config edit and no code.
+
+The distinction that drives all of this is what a mask *means*. An **instrument
+flag** marks a corrupted measurement — the pixels carry no usable signal — so
+these are the only masks that reject anything inside the pipeline. The
+**healsparse masks** are sky-fixed location flags: they say where an object
+sits, not that its pixels are broken, so what to do about one is an analysis
+decision and is made downstream.
+
+**Nothing in the pipeline cuts on the queried columns**, and on exposures the
+query ships off entirely: `MASK_PATHS` is commented out, which makes
+`mask_query` a strict no-op that passes the catalogue through with no
+`MASK_EXT` column (the module stays in the chain, so enabling it is
+uncommenting one line). `star_selection.setools` rejects on `IMAFLAGS_ISO == 0`
+and nothing else, deliberately starting from outlier rejection alone, and the
+final catalogue's `MASK_<band>` columns are written unfiltered. `MASK_EXT` is
+the configurable pickup if outlier rejection proves insufficient: add
+`MASK_EXT == 0` beside each `IMAFLAGS_ISO == 0`, one line per mask block, as
+that file's header documents.
+
+No internet access is needed at any point, and there is no reference star
+catalogue to download.
+
+The one mask that still reaches pixels is the **instrument flag image**
+(`<exp>p.flag.fits.fz`) delivered with each exposure, which records bad columns
+and saturation. `split_exp` splits it per CCD beside the image and weight,
+`sextractor` reads it as `IMAFLAGS_ISO`, and `ngmix` zero-weights flagged
+pixels in its postage stamps. Tiles have no such image, so tile detection runs
+with `FLAG_IMAGE = False`.
 
 ## Detect objects on tiles and process stars on single exposures
 
@@ -328,8 +343,8 @@ Included are galaxy detection and basic measurement parameters, the PSF model at
 galaxy positions, the spread-model classification, and the shape measurement.
 
 Two output directories are created.
-The first one is `run_sp_Ms` for the `merge_sep` run.
-The second is `run_sp_Mc` for the `make_cat` task; the name is the same for both the `MCCD` and `PSFEx` PSF model.
+The first one is `run_sp_tile_Ms` for the `merge_sep` run.
+The second is `run_sp_tile_Mc` for the `make_cat` task; the name is the same for both the `MCCD` and `PSFEx` PSF model.
 
 
 ## Upload results

docs/source/post_processing.md

@@ -9,11 +9,6 @@ catalogue via _metacalibration_), a joint star catalogue, and PSF diagnostic plo
 
 
 
----
-
-If main ShapePipe processing happened at the old canfar VM system (e.g. CFIS v0 and v1), go
-[here](vos_retrieve.md) for details how to retrieve the ShapePipe output files.
-
 ---
 
 ```{note}
@@ -76,6 +71,6 @@ The following steps were used for pre-v1.4 runs performed on the canfar VM syste
       ```
       Choose as input directory `input_dir` the `make_cat` output of the runs
       being combined. A default parameter file `<param_file>` is
-      `/path/to/shapepipe/example/cfis/final_cat.param`. 
+      `/path/to/shapepipe/workflow/config/cfis/final_cat.param`. 
       On success, the file `./final_cat.npy` is created. Depending on the number of
       input tiles, this file can be several tens of Gb large.