Retire in-code catalogue versioning; coverage masks become a workflow product - #880
Draft
cailmdaley wants to merge 37 commits into
Draft
Retire in-code catalogue versioning; coverage masks become a workflow product#880cailmdaley wants to merge 37 commits into
cailmdaley wants to merge 37 commits into
Conversation
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
make_data renders the galaxy with the true PSF and, by default, hands ngmix that same PSF as the model stamp — zero deconvolution error, the one shape systematic the digital twin could not previously probe. Two optional knobs let the model stamp differ from the truth: psf_model_fwhm_ratio (size error) and psf_model_shear (shape error). Defaults reproduce the legacy output byte-for- byte; injected error perturbs only the PSF model stamp, never the galaxy image. Claude-Session: https://claude.ai/code/session_01C9zx7LC72tPMhNGtKnADpp Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* removed tmp files * Moved get_ccds script to pipeline runner and library script * class to download eposure headers (for coverage) * coverage mask: scripts and plotter; using cs_util * coverage code: added Mike Hudson as co-author * documentation (in pipeline_canfar.md) on coverage mask scripts and classes * CCD PSF Handler: Added version for v1.3 * fixed coverage plots * header downloads (coverage): account for existing downloads * canfar-changes * PR: remove non-functioning coverage pipeline function * PR: coverage_map_builder; moved import into try block * PR: field_corners_extractor.py; fixed exp number extraction * PR: added comment * PR: added comment * refactor: consolidate field_corners_extractor helpers; fix regex - Extract _expnum_from_path, _parse_header_to_wcs, _megacam_field_corners as module-level helpers. process_single_header (static, for multiprocessing) and the instance methods now share one implementation. - Remove dead get_wcs_from_header and get_megacam_field (duplicated logic that had already been inlined into process_single_header). - Fix re.search(r'(\d+)\.txt') missing its subject string. The new helper also replaces the hard-coded p[end-6:end] exposure-number extraction in run(), so exposure numbers of any length work. - Declare verbose in params_default so the contract is local instead of relying on cs_util.args.parse_options to inject it. Addresses review feedback on #699. * refactor: coverage_map_builder cleanup - Replace hsp_map + 0 with hsp_map.copy() (healsparse supports .copy since at least 1.x; more explicit intent). - Declare verbose in params_default; currently populated only by cs_util.args.parse_options auto-injection. - Fix typo: "Install te cs_util" -> "Install the cs_util". Addresses review feedback on #699. * docs: declare verbose and annotate survey-specific defaults - coverage_plotter, header_downloader: declare verbose explicitly in params_default. - coverage_plotter: comment acknowledging that FootprintPlotter._regions is a private cs_util attribute; refactor to public accessor when cs_util exposes one. - header_downloader: expand the vos:cfis/pitcairn comment to flag it as UNIONS/CFIS-specific and document override. - build_and_plot_coverage_maps.sh: explain the BUILD_NSIDE=131072 choice (matches UNIONS bit-mask pixel scale, ~0.1"), and note the CoverageMapBuilder default of 2048 for lighter use. Addresses review feedback on #699. * chore: uv lock — register healsparse + skyproj as shapepipe deps The coverage feature adds healsparse (map building) and skyproj (plotting) to pyproject. healsparse was already resolved transitively via cs_util; this pins skyproj 2.5.0 and records both as shapepipe's own direct dependencies so 'uv lock --check' passes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HvP7uKkfDU2PvWC7YmbuMW * fix: coverage console scripts handle -h; drop dead entry point The five coverage/get-ccds runners fell back to `args = sys.argv` (the full argv, including argv[0]) instead of `sys.argv[1:]`, so under the entry-point smoke test's `python -c "...; run_*()" -h` the parser read the program slot as the integer `-c` option and exited non-zero (`invalid integer value: '-h'`). Slice argv[1:] to match cs_util.args.parse_options' own convention. Also drop the `coverage_pipeline` console script, whose target `coverage_run:run_pipeline` was removed in an earlier commit (dangling entry point -> AttributeError on invocation), and declare `hpgeom` explicitly (imported directly by coverage_map_builder, previously only transitive via healsparse). Together these fix all six entry-point test failures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HvP7uKkfDU2PvWC7YmbuMW * test: unit + property tests for the coverage-mask feature Cover the pure and lightly-fixtured logic: exposure-number parsing, the CCD-list -> unique-exposure reduction, the MegaCam field-corner convention, the multi-HDU header -> WCS split, power-of-two nside validation, and a regression guard for the -h/argv handling of the console runners. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HvP7uKkfDU2PvWC7YmbuMW * fix(coverage): subtract missing CCDs in get_ccds_with_psf The v>=1.4 path built the full set of exposure single-HDU IDs but never removed the CCDs whose PSF model is missing, so every candidate CCD was reported as valid. Subtract the missing set so the result is the CCDs that actually have a valid PSF, and report the three counts. * feat(coverage): extract per-CCD corners with optional CCD-list filter Replace the whole-MegaCam-field corner extraction with per-CCD extraction: for each HDU's WCS, build the 4 corners from the CCD pixel bounds (read from pixel_shape, i.e. NAXIS1/NAXIS2) using pixel edges so the quadrilateral covers the full CCD area. Emit one row per CCD keyed on the '<expnum>-<ccd_idx>' ID. Add an optional --ccd_list/-l option: when given, only CCDs in that file (the output of get_ccds_with_psf) are written, so the coverage mask is built from single CCDs with a valid PSF. Resume still keys the done-set on exposure number, now parsed from the CCD-ID first column. Drop the dead _megacam_field_corners whole-field path; keep _parse_header_to_wcs and the multiprocessing structure. * feat(coverage): build maps from per-CCD rows with RA-wrap and pole guards Parse the new per-CCD row format (string CCD ID in column 0, then 8 float corner columns). Add an RA-wrap guard that unwraps polygons straddling the RA=0 seam onto a common branch, and a pole guard that skips polygons reaching |dec| >= 89 deg with a warning (CCD footprints never reach there). Since the CCDs of one exposure do not overlap, accumulating value 1 per CCD polygon still counts exposures-with-valid-PSF per pixel. * fix(coverage): download headers atomically via temp-file rename Copy each header to a '.part' file in the same directory and os.rename on success. An interrupted transfer can no longer leave a partial file that resume treats as complete; the temp file is removed on failure. * docs,build(coverage): align interfaces to the per-CCD flow Fix the console-script name typo (get_ccd_with_psfs -> get_ccds_with_psf) so it matches the docs and script. Harden the build/plot script with 'set -euo pipefail' and document the per-CCD chain (step 0 get_ccds_with_psf feeding extract_field_corners --ccd_list). Update the canfar docs: restore the coverage helpers as shipped, align filenames (ccds_with_psf_<version>.txt, exp_ra_dec_<version>.txt), and describe the per-CCD workflow including step 0. * test(coverage): cover per-CCD extraction, filtering, guards, atomic download Rewrite the coverage tests for the per-CCD design: handler missing-CCD subtraction; per-CCD corner extraction and pixel-shape error; --ccd_list filtering end to end; builder parsing of the new row format; RA-wrap unwrap (a CCD straddling RA=0 fills a small patch, not the complement); pole-guard skip; and atomic-download rename behaviour with a mocked copy. Drop the tests that pinned the removed whole-field convention. * fix(coverage): read true CCD dims from ZNAXIS and resume per-CCD Two correctness fixes in the per-CCD extractor: - Image dimensions: header_downloader fetches fpack tile-compressed headers where NAXIS1/NAXIS2 describe the compressed binary table (byte width, row count), so WCS.pixel_shape reports e.g. (8, 4612) instead of the true (2080, 4612). Read ZNAXIS1/ZNAXIS2 when ZIMAGE is set, falling back to NAXIS1/NAXIS2. _parse_header_to_wcs now returns (wcs, shape) pairs since the WCS drops the Z-keywords; _ccd_corners takes the shape explicitly. - Resume: key the done-set on individual CCD IDs, not exposure numbers. A write interrupted mid-exposure previously left a partial exposure that a -r rerun skipped forever (silent undercount); an expanded --ccd_list rerun skipped exposures whose new CCDs were never written. get_done_ccds reads the present CCD IDs and run() drops only those, so partial exposures are completed and new CCDs added, never duplicating a row. * fix(coverage): single-line CCD list, runner docstrings, plot doc example - header_downloader.get_exposures: wrap the text-format loadtxt in np.atleast_1d so a single-line CCD list (0-d array) does not crash on iteration. - coverage_run: update the extract/build runner docstrings from whole-field 'field corner coordinates' to per-CCD language. - pipeline_canfar docs: add a runnable plot_coverage_map example (the SGC region command) covering the flags build_and_plot_coverage_maps.sh uses. * test(coverage): compressed-header dims, resume path, nexp value contract Extend the coverage tests for the confirmed findings: - fpack-compressed header fixture; assert ZNAXIS dims are recovered and corners span the true CCD width, not the 8-byte compressed NAXIS1. - resume run() path: a new exposure is appended without duplicating existing rows, and a partial exposure (interrupted mid-write) is completed rather than skipped. - the nexp contract on pixel values: two overlapping CCDs from different exposures give value 2 in the overlap and 1 outside. - single-row builder file (atleast_1d/2d guards); single-line get_exposures; RA-wrap strengthened to a positional-parity check (seam CCD vs the same CCD shifted +10 deg agree in pixel count); get_ccds_with_psf subtraction pinned against the real get_all_shdus. * feat(coverage): accept catalogue version v2.0 (9-patch footprint, as v1.6) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MqH9wjbJicVmaQdhZ4kBYr * feat(coverage): v2.0 defaults to 10 patches; -p overrides the version map The v2 footprint is still growing (>5000 deg2 per the 2026-07 Paris meeting, new Dec<30 region); the newest patch definitions (auxdir/CFIS/tiles_202510) include P10. Default v2.0 to 10 patches and add an explicit -p/--n_patch override so the handler tracks evolving patch layouts without code changes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MqH9wjbJicVmaQdhZ4kBYr * fix(coverage): v2.0 is patch-less — read summaries from the run root Martin (PR #812 review): the patch concept is removed in v2.0. For -V v2.0 the handler reads summary/missing_job_32_all.txt and exp_numbers.txt relative to the run root instead of iterating P1..P9; -p restores an explicit P1..P<n> layout when needed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MqH9wjbJicVmaQdhZ4kBYr * docs: state the patch convention per catalogue version (v2.0 is patch-less) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MqH9wjbJicVmaQdhZ4kBYr * feat(coverage): default get_ccds_with_psf -V to v2.0 (patch-less) Align with the collate_star_cat convention (PR #812): v2.0 (patch-less) is now the default catalogue version. Older, patched layouts are selected explicitly with -V. Help text and the pipeline_canfar.md invocations updated; the v2.0 call drops the flag, v1.x shows -V v1.6. Tests set the version explicitly, so none pinned the old default. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WqNFhq9uGpDpprYeuGeb8k * chore: remove accidentally committed pytest log artifact Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * coverage: drop the v2.0 patch-less path — v2.0 coverage moves to the Snakemake stack Martin (review, 2026-09-04): the v2.0 pipeline is Snakemake-orchestrated, has no sky patches, and will not use the summary class or its outputs — so summary/missing_job_32_all.txt and exp_numbers.txt simply won't exist at a v2.0 run root. The patch-less path added in July was built on files that are not going to be there. This PR therefore lands as the v1.x tool. Reverts the v2.0 work (15401d9, a3aa4ec, ae10176, 46214c7, 1ffcd4b) and restores the pre-July contract of CcdPsfHandler: - accepted versions are v1.3, v1.4, v1.5, v1.6 again; -V v2.0 now fails loudly with "Invalid version v2.0" - -V default back to v1.6 (was flipped to v2.0 in 1ffcd4b) - the n_patch = 0 / patches = ["."] "patch-less" semantics are gone; patches are always P1..P<n> from the version map (7/7/8/9) - the -p/--n_patch CLI override is dropped: it was introduced in a3aa4ec purely to let the evolving v2.0 footprint be overridden, and the v1.x patch counts are fixed history Docs lose the "Patch convention by catalogue version" note and the patch-less get_ccds_with_psf invocation; the two v2.0 tests (test_v2_is_patchless, test_n_patch_option_overrides_version_default) are removed. The v1.x version->patch-count parametrisation and the invalid-version test stay. Everything else from the July rework is untouched: per-CCD extraction, RA-wrap and pole guards, atomic downloads, resume, ZNAXIS dims. v2.0 coverage will be a separate PR against the Snakemake stack. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore: relock cs_util to develop 1b15a55 (restores FootprintPlotter.plot_area) --------- Co-authored-by: martinkilbinger <martinkilbinger@cea.fr> Co-authored-by: Cail Daley <cail.daley@cea.fr> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Cail McLean Daley <cailmdaley@gmail.com>
…carriers die nside_coverage=128 / nside=131072 is the production pair that makes the coverage map align pixel-wise with the UNIONS bit masks, and it is NOT the CoverageMapBuilder default (32/2048). Until now those two numbers existed in exactly two places, both slated for deletion in this retirement: scripts/sh/build_and_plot_coverage_maps.sh:25-31 and docs/source/pipeline_canfar.md:545-548. Same for the SGC/NGC plot windows (:41-51 / :555-557) and the sp_validation consumption pointer at :450-452 (notebooks/demo_apply_hsp_masks.py). Transplanted into a commented `coverage:` block in workflow/config.yaml with the bit-mask rationale intact. Commented because nothing reads it yet — the exp_footprint / coverage_map rules land in a later commit; this one only makes sure the deletion cannot take the numbers with it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PJjVNvKrQztrZttuAbbUDW
Both symlink loops (runtime :119-124, dev :170-175) iterate /app/scripts/*/*$ext for ext in .py .sh .bash. scripts/sh/ currently holds exactly seven .bash files and this retirement takes all seven; once the last one goes the glob stays literal, basename '*' '.bash' returns '*', and the build happily creates a junk symlink named '*' in /usr/local/bin instead of failing. `[ -e "$script" ] || continue` makes an empty match a no-op. Lands before any deletion, so no intermediate commit can build the junk link. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PJjVNvKrQztrZttuAbbUDW
`has_option("EXP_BASE_DIR")` selected between two layouts, not two science
modes: the tile branch is the sharded per-exposure store the workflow builds,
the else branch the pre-Snakemake layout where a tile run and its exposures
shared one run root and the header .npy files arrived directly in
input_file_list. The only configs that took the else branch live in
example/cfis/defunct/; every live surface (workflow/config/cfis,
example/cfis/*canfar*, example/cfis_image_sims) sets EXP_BASE_DIR — checked by
grepping every .ini with a [MERGE_HEADERS_RUNNER] section.
Tile mode is now unconditional, so a config that omits the key fails loudly
with NoOptionError instead of silently merging the wrong file list. The
package docstring documents EXP_BASE_DIR and drops OUTPUT_PATH, which the
runner never read, and names find_exposures_runner as the parent (it is the
exp_numbers file that arrives as input; split_exp_runner's headers are reached
through EXP_BASE_DIR).
merge_headers()'s tile_number=None default stays — test_sextractor_post_process
calls it directly.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PJjVNvKrQztrZttuAbbUDW
MULTI-EPOCH mode chose between two ways of finding the psfex_runner output
dirs: ME_DOT_PSF_EXP_DIR resolves them under the sharded per-exposure store,
the else branch parsed 'last:'/'all:' out of ME_DOT_PSF_DIR and asked
log_run_sp.txt which run directory was newest -- the pre-Snakemake
single-run-root layout. The only .ini setting ME_DOT_PSF_DIR are
example/cfis/defunct/config_tile_PiViSmVi{,_canfar}.ini.
Resolving through a run log is exactly what the workflow removed: every rule
writes into one fixed store cleared by unit_pre, so "the newest run_sp_* dir"
has no meaning. Dropping the branch also drops the now-dead
run_log.get_last_dir/get_all_dirs import from this module.
run_log.py itself stays -- mccd_interp_runner:45,91 still calls get_last_dir
(the MCCD chain is a separate, science-level decision).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PJjVNvKrQztrZttuAbbUDW
Same shape as psfex_interp: ME_IMAGE_EXP_DIR + ME_IMAGE_EXP_RUNNERS resolve
each runner's output dir under the sharded per-exposure store, while the else
branch parsed 'last:'/'all:' out of ME_IMAGE_DIR and asked log_run_sp.txt.
Its comment was the only place in src/ that said "v1" outright. The only .ini
setting ME_IMAGE_DIR are example/cfis/defunct/config_tile_PiViSmVi{,_canfar}.ini.
The dropped run_log import leaves get_all_dirs with a single caller in the
tree (scripts/python/get_number_objects.py, itself slated for deletion);
get_last_dir keeps mccd_interp_runner, so run_log.py stays.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PJjVNvKrQztrZttuAbbUDW
Three naming-only strings: ":4 In the v2.0 pipeline", ":43 the example path /arc/home/kilbinger/v2.0/exp", and ":94 "mirrors run_job_canfar_v2.0.sh"" -- a script that no longer exists. The last one is the load-bearing one: it is the only comment explaining WHY the 2-char shard exists, so it now points at the thing that actually builds that tree, rule tile_exp_forest / workflow/scripts/build_forest.py. The globs themselves are untouched -- tile.smk:11-13 and build_forest.py:8-11 depend on exactly this layout. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PJjVNvKrQztrZttuAbbUDW
…nfigs Four comment rewords -- "for [the] v2.0 per-exposure pipeline" -> "for the per-exposure pipeline" in config_tile_PiViVi.ini (workflow and both example/cfis canfar variants) and example/cfis_image_sims'. There is one per-exposure pipeline; naming it after a bash generation only dates it. Also: delete the commented-out CHECK_EXISTING_DIR pointing at a run_sp_Gie_prev dir the workflow never creates (unit_pre clears each stage's own dir), and drop the catalogue-version suffix from example/cfis/config_valjoint_Pl_mccd.ini's FILE_PATTERN (unions_shapepipe_psf_2022_v1.0.2 -> unions_shapepipe_psf_2022). Safe mid-campaign: .ini content is not hashed into any params -- only workflow/scripts/*.py are (Snakefile:316-353) -- and rules name configs by path in their shell string, so a comment edit is not a rerun trigger. Verified none of these files is one of the eleven workflow/config/cfis -> example/cfis symlink targets, and all eleven still resolve. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PJjVNvKrQztrZttuAbbUDW
New localrule `exp_footprint`, one JSON per exposure on the persistent root
giving the four sky corners of every CCD that got a PSF model. It is the raw
material of the campaign coverage mask, and it replaces the v1.x chain's two
approximations with exact local reads of records the workflow already writes:
* WHICH CCDs have a PSF <- exp_persist.json's `files[].name`. Exact, because
psfex_interp returns without writing validation_psf-*.fits on
NOT_ENOUGH_STARS / BAD_CHI2 / FILE_NOT_FOUND. The v1.x tool subtracted a
summary-scrape file from all 40*N candidates.
* WCS <- headers-<exp>.npy, written by exp_split. The v1.x tool downloaded
~25k exposure headers from VOSpace.
Requested by `rule all` directly and NOT gated on `coverage:`, for the reason
persist_targets is: the WCS array is on /scratch and the purge takes it, so
deferring the record until someone wants a map means the map can never be
built. Tombstone-excluded exactly as persist_targets is, and clean_exposure
takes the manifest as an input so reclamation cannot overtake the read.
The rule declares only the durable persist manifest, deliberately not exp_psf's
as well: exp_persist already orders it after the whole PSF chain, and a scratch
manifest in the input list of a durable-output rule would schedule a four-hour
VOS rebuild to satisfy a few KB of provenance.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PJjVNvKrQztrZttuAbbUDW
headers-<exp>.npy names a CCD by its POSITION in the array; exp_persist.json names it inside validation_psf-<exp>-<ccd>.fits. exp_footprint joins the two and nothing cross-checks them, so a permutation would produce a coverage map with the right pixels and the wrong exposure counts, silently. Six synthetic CCDs with distinct WCS centres, three of them PSF-bearing and non-adjacent, so an off-by-one or a "renumber the survivors" bug fails. This test inherits the role of test_coverage.py::test_get_all_shdus, which pinned the same <exp>-<ccd> format against the summary scrape being retired. Also pinned: the record is raw sky across RA=0 (unwrapping belongs to coverage_map_builder, once, at stamping time), a rerun over an unchanged store does not move the manifest's mtime, and the three ways the join can be wrong — a keep list packing no PSF files, a PSF for a CCD the split never wrote, and a persist manifest outliving its scratch store — are loud rather than empty. Corners are cast to plain floats on the way into the record: _ccd_corners hands back numpy scalars, and byte-stability should not rest on json's handling of a float subclass. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PJjVNvKrQztrZttuAbbUDW
New rules/coverage.smk with one campaign-level rule, `coverage_map`, gated by
`coverage: {enabled: ...}` in config.yaml (off by default — the map is a
campaign-END product). It stamps every exposure footprint into
<products_dir>/coverage/coverage.hsp, a HealSparse map counting exposures with a
valid PSF per sky pixel, and writes a manifest naming which exposures went in.
CAMPAIGN-CUMULATIVE. Declared inputs are the in-scope, non-tombstoned
footprints — ordering and rerun semantics without dragging out-of-scope tiles
into the DAG — while the script globs EVERY record on the products root,
reclaimed exposures included. Their records outlive their scratch stores and are
still valid sky, so appending tiles grows the map instead of replacing it.
The production nside pair 128/131072 is now config, not folklore: ~0.1"/pixel,
chosen to match the UNIONS bit-mask resolution so coverage and mask align
pixel-wise. It is NOT the CoverageMapBuilder default (32/2048), and a map built
at the default would look entirely reasonable and would not align. Until now
those two numbers lived only in scripts/sh/build_and_plot_coverage_maps.sh and
docs/source/pipeline_canfar.md, both slated for deletion.
Two parse-time guards, both failing before submission: `coverage:` on with no
validation_psf pattern in `persist_exp:` (there is then no valid-PSF set to
build from), and an nside that is not a power of two.
coverage_map_builder gains the array entry point the workflow needs —
build_map(ccd_ids, ra, dec, ...) plus load_corners() for the text path — so the
CLI and the rule share one implementation of the RA-seam guard, the pole guard
and the polygon accumulation. The exp_ra_dec.txt path is unchanged and still
exercised by tests/module/test_coverage.py.
Plots stay out of the DAG (plot_coverage_map by hand, windows under
coverage.plot), by the argument that keeps run_report.py standalone.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PJjVNvKrQztrZttuAbbUDW
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PJjVNvKrQztrZttuAbbUDW
The runner strips in this branch deleted ME_DOT_PSF_DIR and ME_IMAGE_DIR outright, so a comment saying ME_DOT_PSF_EXP_DIR "replaces ME_DOT_PSF_DIR" now points a reader at a key that raises nothing and does nothing -- the migration note outlived the migration. Say what the key is instead. Comment-only, same four .ini as the previous commit; .ini content is not hashed into any params, so this is not a rerun trigger. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PJjVNvKrQztrZttuAbbUDW
read_footprints already refuses to build from a products root with no footprint records. The other empty map is quieter and worse: records present, every one of them naming zero CCDs. That is a broken PSF stage, not a survey with no coverage, and the .hsp it writes is valid, plausible-looking and masks everything downstream. Fail there too, naming how many records were read. Also names exp_footprint beside exp_persist in run_report's EXP_STAGES comment: it is now the second exposure manifest living on products_dir rather than in the tree this report scans, and the comment is where the next person adding a stage will look. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PJjVNvKrQztrZttuAbbUDW
The v1.x coverage mask was assembled by scraping a finished campaign: read a catalogue version's patch dirs for the CCDs that got a PSF model, download ~25k FITS headers from VOSpace because the run kept no local WCS, and drive the pair from a shell loop over v1.3..v1.6. None of those premises hold. The valid-PSF CCD set is now named exactly by exp_persist.json's tar members, the WCS is already on scratch as headers-<exp>.npy, and the version is the git tag rather than a key the code branches on. The exp_footprint rule reads both and the coverage_map rule stamps the map, so this chain has neither a producer nor a consumer left. Deleted: ccd_psf_handler.py (the version->n_patch table, -V, the P1..P9 loop and the second v1.3 method), header_downloader.py, get_ccds_run.py, its uninstalled duplicate scripts/python/get_ccds_with_psf.py, and build_and_plot_coverage_maps.sh, whose two load-bearing numbers (nside 128/131072, matching the UNIONS bit masks) and SGC/NGC plot windows now live in workflow/config.yaml's coverage: block. The get_ccds_with_psf and download_headers entry points go in the same commit as their modules: test_entrypoints.py parametrizes over that table at import. test_coverage.py loses the tests for the deleted code, including the summary.get_all_shdus assertion that pinned the <exp>-<ccd> ID format -- tests/unit/test_exp_footprint.py now pins that same contract where it is actually relied on. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PJjVNvKrQztrZttuAbbUDW
…und it What the coverage chain was worth keeping is four pieces of survey geometry: the CCD image shape (fpack ZNAXIS vs plain NAXIS), the pixel-edge corner projection, the RA-seam and pole guards, and the nside validation. Everything around them was a way of getting header text files off VOSpace and per-CCD rows into a shared exp_ra_dec.txt — a shape that exists because the pre-Snakemake pipeline kept no per-unit record. The workflow feeds arrays out of the per-exposure exp_footprint.json records instead, so the text format, the resumable directory scrape and the CLIs that drove them have no producer. - field_corners_extractor.py -> ccd_footprint.py, 493 -> 92 lines: the two pure functions the exp_footprint rule calls, and nothing else. - coverage_map_builder.py loses the CoverageMapBuilder class: the text loader, the boolean-threshold derivation and the create_plot branch (plotting has its own entry point and stays out of the DAG). unwrap_ra and build_map are unchanged; check_params' power-of-two validation becomes check_nside, called by build_map so it protects the caller rather than the CLI; median_filter becomes a module function and grows its iteration loop. - coverage_run.py keeps run_plot_coverage alone. extract_field_corners and build_coverage_map leave [project.scripts] in the same commit, per test_entrypoints.py. test_coverage.py is rebuilt around the surviving functions: the header tests construct Headers directly instead of round-tripping through the retired text parser, and the map tests call build_map instead of writing a corners file and running the CLI. The nexp contract, the RA-shift invariance guard and the pole guard are unchanged in substance. Building the map no longer writes a stray `log_-i` into the working tree, since cs_util's log_command went with the CLI. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PJjVNvKrQztrZttuAbbUDW
summary.py reconstructed a campaign's state by walking a finished run's patch
directories: a five-part path recipe, newest-run_sp_* resolution, n_CCD = 40
hardcoded, bit-named outputs (missing_job_<bit>_all.txt) and an eight-code
error-signature whitelist driving a resubmission loop. summary_params_pre_v2.py
was its parameter table, and every line of it is a per-patch branch, a -j bit
key or a canfar-era run-dir name — `n_sh = 1 if patch in ("P2","P5","P8","P9")
else 8`, a run root under $HOME/cosmostat/v2/pre_v2, a hand-appended workaround.
The workflow does not scrape. Each rule records its own completeness as it goes
(completeness.py's per-stage floors and reason scrape, written to
manifests/<stage>.json) and run_report.py tallies those records, so the state of
a campaign is read rather than inferred. The patch is not a concept either:
scope is a tile list, and summary_tiles.py's hardcoded `n_patch = 8` — bumped
7->8 in 2025-01 and never bumped for P9 — is what that assumption costs.
Also deleted: summary_run.py and its bin/ shim, summary_tiles.py, and
summary_run.ipynb, which has been dying at cell 3 since summary_params_pre_v2
moved into the package and broke its `%run` path. The summary_run entry point
goes in the same commit as the module (test_entrypoints.py), and CLAUDE.md's
entry-point list is updated to what actually ships.
The last importer was ccd_psf_handler.py, retired one commit ago; nothing in
src/, workflow/ or tests/ imports summary now.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PJjVNvKrQztrZttuAbbUDW
Also sorts coverage_map_builder's import block, which the CLI's cs_util imports had been hiding. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PJjVNvKrQztrZttuAbbUDW
The 2020-2022 CANFAR VM / HTCondor layer, superseded by the Snakemake workflow and already broken in several places: - canfar_submit_selection.sh writes HTCondor submit files naming job_sp.bash, which has not existed in the tree for years. - canfar_download_results.bash pulled per-tile .tgz back from vos:cfis/cosmostat/kilbinger/results; products now land on products_dir directly. - post_proc_sp.bash chains combine_runs -t psf (an option combine_runs never parsed) and untar_results.sh (absent). - canfar_avail_results.py counted an expected result-name set off a vos listing; `sp report` reads the manifests instead. - stats_headless_canfar.py printed its own deprecation notice and was the sole importer of skaha; canfar_monitor -q replaces it. - merge_final_cat.py was superseded by create_final_cat.py per its own docstring, and its input was a combine_runs.bash link farm. skaha therefore loses its last importer: dropped from pyproject, from the dependencies table, and (surgically, to avoid an unrelated marker rewrite from a newer uv) from uv.lock. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PJjVNvKrQztrZttuAbbUDW
vos_retrieve.md declared itself obsolete on its first line and was an orphan (never in toc.rst), so deleting it also clears a standing Sphinx warning. post_processing.md documented the pre-v1.4 canfar-VM chain and named create_sample_results, which does not exist; PSF validation and the rho-statistics live in sp_validation now. random_cat.md keeps the module, which is current, and stops calling tools that no longer ship: the mask-staging and mask-download steps become a statement of what random_cat_runner needs on input, and the joint-catalogue step says plainly that merge_final_cat (retired at 2ef07e4) has no successor and the concatenation is a hand step. The header-download step stays: config_get_tiles_vos_headers.ini is live. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PJjVNvKrQztrZttuAbbUDW
The Gen 1 generation — CANFAR headless sessions driving the `tile_runs/`
/ `exp_runs/` layout — is what the Snakemake workflow replaces rule for
rule, and much of it was already broken:
- curl_canfar_local.sh called call_curl with 14 args against a
10-parameter function, and polled `stats_jobs_canfar.sh -w all`, an
option that script never parsed.
- init_run_exclusive_canfar.sh was the run layout in one file: the
`${kind}_runs/$ID` roots, the symlink pull-in, the bit codes. The
rules in prepare/exposure/tile.smk plus the Snakefile's unit_pre and
sp_shell carry all of it now.
- canfar_async_job.sh was already unreachable: run_async always
selected async_bulk.
- run_scratch_local.sh was orphaned and broken (it guards $exclusive
after unconditionally assigning $ID).
- combine_runs.bash built a post-hoc symlink farm; tile_make_cat and
exp_persist publish to products_dir instead.
- remove_duplicates_tiles.sh cleaned up duplicate timestamped run dirs,
which one-rule-one-store makes structurally impossible.
- stats_jobs_canfar.sh parsed skaha JSON by grep line offsets;
canfar_monitor -q does it through the API.
- distribute_tiles.py and link_to_exp_for_tile.py chunked and globbed
what run_index.sqlite and build_forest.py now index; the latter was
already dead (its only caller commented it out for P9).
- clear_ngmix_prev.py had zero references anywhere.
- get_number_objects.py's count floors live in completeness.py.
canfar_submit.py goes with them: every mode it can select execs one of
these three scripts (async_single → canfar_async_job.sh, async_bulk →
distribute_tiles.py, sync → init_run_exclusive_canfar.sh), so the head
of the chain has nothing left to submit. Its console entry point and
bin/ shim go too. canfar_monitor and canfar_log_monitor stay — they are
layout-independent session management, and the substitute for the
stats script deleted here.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PJjVNvKrQztrZttuAbbUDW
run_job_sp_canfar_v2.0.bash — still driven by sp_validation's image-sims workflow — sources this file for `command()` and nothing else. Everything around it belonged to the curl/skaha submission layer deleted in the previous commit: the CANFAR session URL, the images.canfar.net tag, `version="2.0"`, `cmd_remote`, `call_curl` (whose 10 parameters the one caller passed 14 arguments to), `update_session_logs`, `get_line_from_file` (REPLICA_ID fan-out) and `get_kind_from_job` (bit codes). `command()` is byte-identical. `pat` and `STOP` stay: `command()` reads both as free variables, and this file has always been the one that set them for run_job_sp_canfar_v2.0.bash. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PJjVNvKrQztrZttuAbbUDW
pipeline_canfar.md was the v1.x production procedure end to end: the `patch=P[1-9]` loop, `-V v1.6`, the bit-coded `-j` table, and every script deleted in this PR. Its two load-bearing pieces were transplanted in Phase 0 before anything was removed — the 128/131072 healsparse pair with its "matches the bit masks" rationale, and the SGC/NGC plot windows, now in workflow/config.yaml's coverage block, plus the sp_validation consumption pointer (notebooks/demo_apply_hsp_masks.py). clusters.md keeps candide and ccin2p3 and replaces its CANFAR section with what a reader needs to know: submission is retired, orchestration is the Snakemake workflow, that workflow is SLURM-only, and the two monitor commands survive. Also fixes job_sp_canfar_v2.0.bash's header, which named the Gen 1 script deleted here rather than itself. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PJjVNvKrQztrZttuAbbUDW
Both lines described a capability this PR removed. The `canfar` package stays, but its only importers are now canfar_monitor and canfar_log_monitor — session monitoring, not job submission — and clusters.md no longer carries a worked CANFAR example. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PJjVNvKrQztrZttuAbbUDW
…ng else The script had three axes of branching on top of one collation: the catalogue version (-V, with the v1.3..v1.6 patch table), the sky patch (-P, the P<n> loop and the patch token in the output filename), and the PSF model (-p, psfex or mccd). Only one combination is used and only one is buildable: -V v2.0 -p psfex, i.e. patch-less input under <input_base_dir>/output and psfex_interp outputs. Deleted, in order of how dead they were: - version_cat/-V, allowed_versions, patches/-P, collate_paths(), the patch loop, the mode=="test" P3/P4 patch list, and the patch parameter of output_filename()/transform_exposures(). Names lose the patch token they already lacked at -V v2.0: validation_psf_conv-<idx>.fits, still what example/cfis/config_Ms_psfex_conv.ini's FILE_PATTERN consumes. - the psf=="mccd" branch. It is unreachable in practice and broken where it is reachable: no workflow config names an mccd runner, and its header path concatenated "setools_runner/output/mask" with "validation_psf" without a separator, a failure the bare `except: continue` swallowed silently. With it go transform_shape() (its only user, and hence the galsim import), the vendored Loc2Glob/Glob2CCD focal-plane geometry (its only users), _dt_mccd and sub_dir_setools. transform_shape's math is not lost: tests/module/test_hsm_sky_coords.py already carries it verbatim as the reference the world-coordinate HSM measurement is checked against. - update_params(), whose whole body was the psf switch. The two surviving paths are module constants (EXP_RUN_GLOB, PSF_INTERP_SUBDIR). tests/module/test_collate_star_cat.py goes in the same commit: every test in it pinned patch versus patch-less, a distinction that no longer exists. The module docstring gains the caveat this strip must not be read as closing: no workflow rule produces validation_psf_conv-*.fits. The input layout here is still a link farm of psfex_interp run directories, while the workflow persists validation_psf-<exp>-<ccd>.fits into the exp_persist tars. Repointing the script at those is a follow-up. 895 -> 301 lines, with the psfex collation block byte-identical modulo indentation. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PJjVNvKrQztrZttuAbbUDW
`grep -rni snakemake docs/source` returned zero hits: the one production orchestration was documented only in workflow/README.md, which nothing in the Sphinx tree linked. This adds docs/source/workflow.md under "Running on a cluster" -- quick start, the two static invocations and why the exposure set cannot live in the tile DAG, manifests as the DAG's success-only currency, `sp report`, `sp container`, the two roots and the durable products, and the coverage chain -- and points at workflow/README.md for the deep reference. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PJjVNvKrQztrZttuAbbUDW
Every runnable command on the introductory tutorial was dead: nine invocations of `job_sp -j <bit>`, and `scripts/sh/job_sp.bash` has not existed for some time. The bit-coded spine is replaced by the workflow's rules, stage by stage, each section now naming the rule that does the work and the config it runs. Kept unchanged: the file-type, naming and numbering conventions (the most durable content in docs/source) and all the diagnostic commentary -- the ds9 flag comparison, the star-selection plots, the star_stat example, the stats_global histograms. Also corrected while passing through: masking now says there is no tile_mask rule (the committed chain is the unmasked tile_detect variant) and that the star catalogues come from the exp_star_cat cache rather than being staged by hand; the MCCD alternatives are dropped from the PSF and galaxy sections, since no workflow config names an mccd runner. The "Upload results" section goes with the `-j 128` step it documented. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PJjVNvKrQztrZttuAbbUDW
Three things had drifted apart from the code: - The opening sentence spoke of the bash layers in the present tense, as something this workflow "replaces". They are retired; say so, and say plainly that this is the one production orchestration. - `sp_rule.py` does not exist and never did on this branch. The README named it in the Layout block, in completeness.py's description and in the count-floor bullet, and config_exp_Sp.ini cited it for the dashed-ID convention. The real mechanism is the Snakefile's `unit_pre()` (the furniture and the run-dir clear, inlined as params bash) and `sp_shell()` (prologue, one shapepipe_run, one `completeness.py check`); the dashed ID comes from `unit_num()`. - The Layout block omitted three scripts that are there: star_cats.py, ngmix_range.py, clean_tile.py. Adds the one line the retirement turns on: no catalogue version in the code, no sky patches, git tags are the version. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PJjVNvKrQztrZttuAbbUDW
…ripts/ CLAUDE.md's Layout block had no entry for `workflow/` at all -- the one production orchestration was invisible to anyone reading the repo map. Adds it, naming bin/sp, the rule files, the scripts, the config chain and the profile, and adds the sentence the retirement turns on so the decision does not have to be rediscovered: no catalogue version in the code, no sky patches, git tags are the version. README.rst gains a workflow bullet beside the cluster one. scripts/README.rst described two scripts, one of which (`create_log_exp_headers`) has not existed for years, and gave a usage line for the other that the script's own --help supersedes. Rewritten as a map of what is actually in the directory, grouped by what the scripts are for, with the container's bare-name symlinking stated once. It deliberately does not restate any script's interface -- that is what drifted. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PJjVNvKrQztrZttuAbbUDW
`#select-stars`, `#download-tiles` and `#find-exposures` named headings this page has never had. Point them at the sections that actually describe those steps. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PJjVNvKrQztrZttuAbbUDW
The tutorial's in-page links (`[Mask images](#mask-images)` and friends) each raised a `myst.xref_missing` warning on every build, because MyST emits no heading targets unless asked for them. `myst_heading_anchors = 3` covers h1-h3, which is every heading those links point at, and clears five warnings. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PJjVNvKrQztrZttuAbbUDW
workflow/README.md, docs/source/workflow.md and CLAUDE.md each asserted that the bit-coded bash job layer was retired at 2ef07e4 along with the CANFAR front end. The CANFAR half is true; the bash half is not. scripts/sh/ still carries run_job_sp_canfar_v2.0.bash, job_sp_canfar_v2.0.bash, job_list_help.bash and functions.sh, pyproject's [tool.setuptools] script-files still installs the first two, and sp_validation's image-sims workflow calls run_job_sp_canfar_v2.0.bash by path — a live cross-repo dependency a reader of those three pages would have concluded was gone. scripts/README.rst already said this correctly. Each page now names what actually went and why the pair stays. The Gen-2 scripts carry no catalogue-version or patch constructs (verified by grep), so keeping them is consistent with the decision; they retire when the image-sims chain is ported or parked. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PJjVNvKrQztrZttuAbbUDW
`uv lock` (uv 0.11.28, the dev image) after dropping skaha. The only package-level change is skaha gone; the rest is uv dropping the redundant `sys_platform == 'linux'` markers, which the next Renovate lockFileMaintenance would do regardless. Replaces the hand edit in 9ee5732. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PJjVNvKrQztrZttuAbbUDW
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Retire in-code catalogue versioning; coverage masks become a workflow product
Branch:
feat/retire-versioning→ basefeat/persist-exp-products(#879) while it is unmerged, thenfeat/snakemake-orchestration(#852). Containsdevelopat #797.Size: 31 commits, 82 files, +2 069 / −11 219. Every deleted file was last carried at
2ef07e45(git show 2ef07e45:<path>).Why
ShapePipe writes shear catalogues; it never reads, reprocesses, or has to stay compatible with a v1.x run. So the code has no business knowing which catalogue version it is producing — git tags are the version. Until now it carried the v1.x era inside it: the
v1.3/v1.4 → 7,v1.5 → 8,v1.6 → 9patch tables,-V/-Pflags,P1..P9loops, the post-hocsummary/+missing_job_*scrape, and three generations of CANFAR bash whose job the Snakemake workflow already does. After this PR no code path branches on a catalogue version, and the constructs that existed only for that layout are gone with it.The
src/half turned out small:pipeline/andmodules/contained no catalogue versioning — only three dual-layouthas_optionswitches (EXP_BASE_DIR,ME_DOT_PSF_EXP_DIR,ME_IMAGE_EXP_DIR) whose legacy branches were reachable fromexample/cfis/defunct/alone. The mass is the pre-Snakemake run infrastructure.Coverage masks as a workflow product
#797 landed the v1.x coverage tool (scrape patch dirs → download ~25k headers from VOSpace → subtract
missing_job_32_all.txt→ appendexp_ra_dec.txt). Every move there existed because v1.x kept no per-unit record. The workflow keeps two, so:exp_footprint(per exposure, localrule, afterexp_persist): for every CCD with avalidation_psf-<exp>-<ccd>.fitsmember inexp_persist.json— exact, sincepsfex_interpwrites that file only on success — take the WCS fromheaders-<exp>.npyand write the four sky corners to<products_dir>/exp/<shard>/<exp>/manifests/exp_footprint.json(byte-stable, tmp→cmp→mv likepersist_exp.py).clean_exposuretakes it as an input so reclamation cannot overtake the read. The CCD-index invariant (npy index i ==<exp>-<i>==validation_psf-<exp>-<i>.fits) is pinned bytests/unit/test_exp_footprint.py, inheriting the role of the oldsummary.get_all_shdustest.coverage_map(campaign-level,workflow/rules/coverage.smk, off by default viacoverage.enabled): declared inputs are the in-scope non-tombstoned footprints; the script reads every record on the products root, tombstoned included, so the map is campaign-cumulative.nside_coverage=128 / nside=131072(matching the UNIONS bit masks) now live only inconfig.yaml; a parse-time guard fails the launch ifpersist_exp:no longer packsvalidation_psf-*. Plots stay out of the DAG (plot_coverage_mapby hand, windows inconfig.yaml)._image_shape,_ccd_corners, RA-seam and pole guards, nside validation);CoverageMapBuilder,get_ccds_with_psf,header_downloader,build_and_plot_coverage_maps.share retired.Not yet done: the acceptance run — build a map on a real campaign and compare against a v1.6
.hspover the overlap. Needs a campaign withexp_persistproducts (nibismk-g6).Retired / replaced
utilities/ccd_psf_handler.py,get_ccds_run.py,scripts/python/get_ccds_with_psf.py-V, v1.x→n_patch,P{x}loop,ccds_with_psf_<version>.txt)exp_footprintruleutilities/header_downloader.pyheaders-<exp>.npyalready on scratchutilities/field_corners_extractor.py→ccd_footprint.pyscripts/sh/build_and_plot_coverage_maps.shVERSIONS=("v1.3" … "v1.6")loopcoverage_maprule +plot_coverage_maputilities/summary.py,summary_params_pre_v2.py,summary_run.py,scripts/python/summary_tiles.py,summary_run.ipynb,docs/post_processing.mdn_patch = 8hardcoded (never bumped for P9)manifests/<stage>.json+logs/<stage>.json,completeness.py,sp report(run_report.py)scripts/python/collate_star_cat.py(stripped)-V/-P,allowed_versions, patch loop, unreachable mccd branch-V v2.0default path, verbatimcanfar_submit_selection.sh,canfar_download_results.bash,post_proc_sp.bash,canfar_avail_results.py,stats_headless_canfar.py,merge_final_cat.py,docs/vos_retrieve.mdproducts_dircurl_canfar_local.sh,init_run_exclusive_canfar.sh,canfar_async_job.sh,run_scratch_local.sh,combine_runs.bash,remove_duplicates_tiles.sh,stats_jobs_canfar.sh,job_sp_canfar.bash,distribute_tiles.py,link_to_exp_for_tile.py,clear_ngmix_prev.py,get_number_objects.py,canfar/canfar_submit.py,docs/pipeline_canfar.md${kind}_runs/$IDlayout, bit-coded job masks, symlink pull-in ofrun_sp_*, patch-specific special cases (P8/P9 -m 1 -s 1)prepare/exposure/tile.smk,unit_pre()/sp_shell(),tile_exp_forest,run_index.sqlite,canfar_monitormodules/{merge_headers,psfex_interp,vignetmaker}_runner.py(stripped)log_run_sp.txt(last:/all:)EXP_BASE_DIR/ME_*_EXP_DIRfails loudlyskahadependencycanfarstays:canfar_monitor,canfar_log_monitor)Container
$PATHcommands that disappear (the Dockerfile symlinks everyscripts/*/*):summary_run,get_ccds_with_psf,download_headers,extract_field_corners,build_coverage_map,canfar_submit_job,summary_tiles,build_and_plot_coverage_maps,canfar_avail_results,stats_headless_canfar,merge_final_cat,distribute_tiles,link_to_exp_for_tile,clear_ngmix_prev,get_number_objects, and the gen-0/1 shell names. The Dockerfile's glob loops are now guarded against an empty match.Deliberately kept, and why
run_job_sp_canfar_v2.0.bash,job_sp_canfar_v2.0.bash,job_list_help.bash,functions.sh→command()only,update_runs_log_file.py,init_run_v2.0.py, the[tool.setuptools] script-filesblock): sp_validationdevelopworkflow/rules/image_sims.smkcallsrun_job_sp_canfar_v2.0.bash. It carries no version or patch construct — it survives on a cross-repo dependency, not a versioning exemption — and retires when the image-sims chain is ported to this workflow (example/cfis_image_sims/README.mdis the porting spec).create_final_cat.py: writes the HDF5 grouppatches/<P n>thatsp_validation/catalog.pyandcatalog_builders.py(with their own v1→7 / v1.5→8 table) read. Renaming lands in lockstep with an sp_validation issue; its docstring still shows-P 7.completeness.py,ngmix_range.py,tile_local(): hashed intoparamson every rule — a comment edit replans a whole campaign. Their provenance comments pointing at deleted paths get reworded at a campaign boundary.canfar_monitor/canfar_log_monitor: layout-independent account tools, and the substitute for the deletedstats_jobs_canfar.sh.Questions for review
canfar_submit.pyis gone; the workflow has no CANFAR execution mode). Is anyone still launching production on CANFAR? If so the last commit carrying it is2ef07e45, and a CANFAR profile for the workflow is the path back.check_tile_coverage.pysays sp_validation calls it; no checkout we can see does. Kept — is it live?merge_final_cat.pywas the only merger for per-tile random catalogues (random_cat.mdnow says the concatenation is a hand step). Is a joint-randoms product needed for the tomographic round?Also in this PR
Docs:
docs/source/workflow.md(new),pipeline_tutorial.mdrebuilt on the rule chain (its ninejob_spinvocations named a script that did not exist),workflow/README.mddrift fixed (sp_rule.pynever existed; the mechanism isunit_pre()/sp_shell()+completeness.py check).uv.lockis regenerated from pyproject (uv lock, uv 0.11.28 in the dev image): the only package-level change isskahagone; the rest of the diff is uv dropping the now-redundantsys_platform == 'linux'markers, which the next Renovate lockFileMaintenance would do regardless.Tests:
tests/unit+tests/module/test_coverage.py+test_coverage_map.py+test_sextractor_post_process.pygreen in the dev image; one pre-existing failure elsewhere (test_metacal_psf_azgauss_is_a_live_alternative, an ngmix/galsim assertion in the container).Follow-ups (named, not done)
Image-sims port off the gen-2 bash; MCCD tile chain is unreachable from the workflow (no
config_tile_MiViVi_*exists) — retiring it freesrun_log.get_*_dirs; ~30 decoratorversion="X.Y"strings never bumped or compared;collate_star_cat.pystill expects a psfex_interp link farm rather thanexp_persisttars;build_index.py's deadra_dircolumn needs a sqlite migration; notile_maskrule exists (the tutorial used to describe one);example/config.iniis the toy example, not "a single CFIS tile" asCLAUDE.md/README.rstclaim.🤖 Generated with Claude Code