Snakemake orchestration for the real-data pipeline - #852

Open

cailmdaley wants to merge 105 commits into

feat/snakemake-orchestration

Commits on Jul 16, 2026

feat(orchestration): Snakemake workflow for the real-data pipeline (#848)

cailmdaley
and
claude
committed
f92cb44
fix(orchestration): resolve {threads} at rule level, not inside params.cmd

cailmdaley
and
claude
committed
187950c
fix(orchestration): carry original exposure names through the index; star-cat store naming

cailmdaley
and
claude
committed
ac6bddd
fix(orchestration): exposure-name pass-through, star-cat dirs, P0 run config

cailmdaley
and
claude
committed
9b0456e
fix(orchestration): {output} placeholder inside params.cmd survives literally

cailmdaley
and
claude
committed
6fe141f
fix(orchestration): shard the exposure forest — the module glob's shape is not cosmetic

cailmdaley
and
claude
committed
a110d89
fix(orchestration): carry a wrapper-scripts content hash into every rule's params

cailmdaley
and
claude
committed
28c10d8
fix(orchestration): keep snakemake state and index off /project (hard-full quota)

Cail Daley
and
claude
committed
a273e76

Commits on Jul 17, 2026

fix(orchestration): drop exp_mask's dead per-unit star-cat input

Cail Daley
and
claude
committed
0010def

Commits on Jul 31, 2026

fix(orchestration): nibi-flavor completeness counts; file-level attrition reporting

Cail Daley
and
claude
committed
833aa93
feat(ngmix): env-expanded chunk fields; position-seeded RNG becomes the only mode

Cail Daley
and
claude
committed
20cc506
feat(orchestration): committed workflow configs — no wrapper, no runtime rewriting (#848 D2)

Cail Daley
and
claude
committed
d1c0afb
feat(orchestration): manifests become the DAG's currency — completeness check CLI + manifest-driven report (#848 D2/D3)

cailmdaley
and
claude
committed
1b22b98
feat(orchestration): the rules rewrite — manifests as DAG currency, sharded stores, parse-time index, native scattergather (#848 D1–D5)

cailmdaley
and
claude
committed
83d26f2
fix(orchestration): harden the rewrite against its adversarial review

cailmdaley
and
claude
committed
b149e18
feat(orchestration): clean_exposure becomes an in-DAG rule with tombstones (#848 D5, S5)

Cail Daley
and
claude
committed
12ecfc2
feat(orchestration): nibi profile goes slurm-executor (#848 D4, S6)

Cail Daley
and
claude
committed
e677f67
fix(orchestration): stop the clean_exposure cascade; report reads tombstones (#848 D5)

Cail Daley
and
claude
committed
96c78cd
docs(orchestration): profile comment tells the truth about the clean/trigger gate

Cail Daley
and
claude
committed
4d28897
run: point the quad at a fresh smk-g3 run dir (old smk-p0 tree is unsharded, pre-manifest)

Cail Daley
and
claude
committed
831a127
fix(orchestration): slurm_account is the association name (def-mjhudson_cpu)

Cail Daley
and
claude
committed
f82e7f4

Commits on Aug 15, 2026

fix(orchestration): index build is the main compute parse's alone

Cail Daley
and
claude
committed
5cfdc04
fix(orchestration): sp pins --snakefile; works from any cwd

Cail Daley
and
claude
committed
acfb91a
docs(orchestration): a failed tile pins ~8 exposures, not ~80

Cail Daley
and
claude
committed
dedc1f0
fix(orchestration): manifests are success-only DAG currency

Cail Daley
and
claude
committed
2a24d34
feat(orchestration): star-cat creation moves into the DAG

Cail Daley
and
claude
committed
a1d4dfa

Commits on Aug 16, 2026

fix(orchestration): the verdict composes counts with shapepipe_run's rc

Cail Daley
and
claude
committed
6a8e71a
fix(orchestration): exp_star_cat stops trusting what it cannot see

Cail Daley
and
claude
committed
7341012
fix(orchestration): reclaim the star-cat farm, stop the home-quota leak

Cail Daley
and
claude
committed
cba14de

Commits on Aug 24, 2026

run(smk-g3): flip clean on — quad validated, batch1 append needs the room

Cail Daley
and
claude
committed
30567ac
run(smk-g3): the campaign appends the first 30 batch1 tiles

Cail Daley
and
claude
committed
fadc277

Commits on Aug 28, 2026

Move failure evidence onto snakemake's native log: directive

cailmdaley
and
claude
committed
93c7b4e

Commits on Aug 29, 2026

Fetch star catalogues by sky chunk, not by exposure

cailmdaley
and
claude
committed
b2cfa3b
feat(container): sp container — per-user image layers and resolution

cailmdaley
and
claude
committed
f2cf65b
Fuse the short rules into per-unit group jobs

cailmdaley
and
claude
committed
1ed2537
Derive merge_sep_cats chunk paths from the run-directory name

cailmdaley
and
claude
committed
aee48dc
Fail loudly on an unset SP_PHASE and an unresolvable container

cailmdaley
and
claude
committed
d5c208e
Group tile_exp_forest with tile_merge_headers

cailmdaley
and
claude
committed
15c4e6c
fix(psf): deterministic star train/validation split; align tile STAR_THRESH with the 80% convention

authored and
cailmdaley
committed
109739d
fix(configs): the workflow's own PiViVi config takes STAR_THRESH 22 too

cailmdaley
and
claude
committed
9078209
feat(orchestration): the persistent root D5 always described

cailmdaley
and
claude
committed
43987f6

Commits on Aug 30, 2026

feat(orchestration): the shape chain becomes one node-local job

cailmdaley
and
claude
committed
a0311c2
fix(orchestration): derive the node-local path, do not try to pass it

cailmdaley
and
claude
committed
fb28e7a
docs(profile): the $SLURM_TMPDIR passthrough is disproven, say so

cailmdaley
and
claude
committed
7b34c9e
fix(orchestration): tile_ngmix runtime scales with attempt

cailmdaley
and
claude
committed
6224206
perf(orchestration): stage the WCS sqlite node-local with the vignette store

cailmdaley
and
claude
committed
b2163e0
perf(orchestration): tile_ngmix mem_mb 14000 -> 5000, measured across 31 tiles

cailmdaley
and
claude
committed
af772ea
fix(orchestration): make the WCS staging atomic; correct two memory claims

cailmdaley
and
claude
committed
7302a2d
run(smk-g5): a one-tile paired control for the two unvalidated fixes
cailmdaley
committed
234b2c3
perf(orchestration): split ngmix chunks by epoch-weighted cost, not object count

cailmdaley
and
claude
committed
a8ec764
feat(orchestration): reclaim a finished tile's scratch store

cailmdaley
and
claude
committed
4f389ea
Merge branches 'dev/chunk-balance' and 'dev/clean-tile' into dev/gate3-merged
cailmdaley
committed
dc9101a
docs(orchestration): range_hash is safe to add only at a campaign boundary

cailmdaley
and
claude
committed
f196fa1
fix(report): reclamation must not edit a tile's verdict

cailmdaley
and
claude
committed
731f8cc
docs(orchestration): say which half of the range_hash hazard was observed

cailmdaley
and
claude
committed
cfe62ed
docs(orchestration): mark the Snakefile half of the deletion claim as derived

cailmdaley
and
claude
committed
503c083
run(smk-g6): point the campaign at a fresh 64-tile root
cailmdaley
committed
4b96eff
docs(orchestration): the final_cat deletion is now observed, not derived

cailmdaley
and
claude
committed
acaeb0b
Merge origin/develop: uberseg blend handling + WCS-derived pixel scale onto position-seeded RNG

cailmdaley
and
claude
committed
5daa82c
refactor(smk): one script_hash helper, unit_pre derives its level, star-cats use the resolved image

cailmdaley
and
claude
committed
fd91365
refactor(smk): sp_shell composes the two special tile bodies

cailmdaley
and
claude
committed
9ca1800
perf(scripts): drop dead stage-table rows, bound the report's globs, stat less

cailmdaley
and
claude
committed
8610c0a
refactor(star-cats): one atomic write and one focal-plane geometry, in the library

cailmdaley
and
claude
committed
725a8b0
chore(launcher): parse config.yaml with yaml, flag the profile's one non-portable line

cailmdaley
and
claude
committed
b158cb9
workflow/config/cfis: symlink the 11 byte-identical data files to example/cfis

cailmdaley
and
claude
committed
07f799d
docs(smk): one home per concept in the Snakefile, rules and profile

cailmdaley
and
claude
committed
db95741
docs(scripts): point the reclamation and report scripts at their one home

cailmdaley
and
claude
committed
928297e
docs(ngmix): position_seed owns the position-seeded RNG story

cailmdaley
and
claude
committed
c2763f8
feat(ngmix): materialise the chunk partition once per tile

cailmdaley
and
claude
committed
8726e9b
feat(sp): snapshot the code at launch so edits cannot reach a running campaign

cailmdaley
and
claude
committed
bedf996
ngmix: hoist per-object sqlitedict reads out of the epoch loop

cailmdaley
and
claude
committed
84435c4

Commits on Aug 31, 2026

feat(mask): declare healsparse + hpgeom dependencies

cailmdaley
and
claude
committed
cad7eb5
feat(mask_ext): rasterize external healsparse masks to pipeline flag images

cailmdaley
and
claude
committed
3bb39e3
feat(make_cat): per-band MASK_<BAND> columns from external healsparse maps

cailmdaley
and
claude
committed
6baa474
feat(mask_ext): fail fast on boolean masks with non-1 bits

cailmdaley
and
claude
committed
ba20d53
Add MASK_EXT_PATHS example to make_cat config

cailmdaley
and
claude
committed
f554e2b
refactor(mask): delete internal mask generation and its star catalogues

cailmdaley
and
claude
committed
eb93838
feat(mask_query): flag exposure detections against external healsparse masks

cailmdaley
and
claude
committed
659168a
refactor(pipeline): rewire the chains around the query design

cailmdaley
and
claude
committed
51f25a5
docs(mask): describe the query design, drop the deps only masking used

cailmdaley
and
claude
committed
1bceb2d
feat(mask_query): narrow the shipped PSF-star diet to the star-body map

cailmdaley
and
claude
committed
897febc
fix(mask): address review — CI smoke, SEGMENTATION checkimage, canfar docs

cailmdaley
and
claude
committed
ba2271a
perf(mask_query): read only the coverage pixels a catalogue touches

cailmdaley
and
claude
committed
f7d1fd6
revert(scripts): restore collate_star_cat — it is PSF validation, not masking

cailmdaley
and
claude
committed
2cc82ee
fix(mask_query): public sentinel, empty-coverage guard, real product name

cailmdaley
and
claude
committed
ed376f7
refactor(random_cat): delete — randoms come from healsparse map algebra downstream

cailmdaley
and
claude
committed
c7c9e82
fix(config): the 2026 DR6 products are named mask_ugriz_* — point the placeholder at the real ladder

cailmdaley
and
claude
committed
edec546
feat(mask_query): ship permissive — flag PSF-star candidates, cut nothing by default

cailmdaley
and
claude
committed
97c44b7
feat(mask_query): ship the query off by default, rename FLAG_EXT to MASK_EXT

cailmdaley
and
claude
committed
43c32cd

Commits on Sep 5, 2026

ngmix: clarify position seed rationale
cailmdaley
committed
d5c73a1
ngmix: remove obsolete seed configuration
cailmdaley
committed
899763a
workflow: rename tile merge catalogue config
cailmdaley
committed
6890d8a
workflow: standardize tile catalogue run names
cailmdaley
committed
d82f214
workflow: simplify explanatory prose
cailmdaley
committed
cbf7e11
tests: mark UNIONS-specific tile pruning test
cailmdaley
committed
4ceb91a
workflow: centralize configured inputs and outputs
cailmdaley
committed
455583b
workflow: select PSF model from configuration
cailmdaley
committed
98bc085
Remove completeness count floors
cailmdaley
committed
4e130a8
clean_exposure: update exp_psf run name in docstring
cailmdaley
committed
c7509af

Commits on Sep 6, 2026

Commits on Sep 9, 2026

example/cfis: retire configs superseded by the Snakemake workflow

cailmdaley
and
claude
committed
e32c4fb
docs: add Snakemake workflow page, drop obsolete VOspace retrieval page

cailmdaley
and
claude
committed
3840fee
scripts: remove unreferenced canfar/vos helper scripts

cailmdaley
and
claude
committed
7e2036d
Merge remote-tracking branch 'origin/develop' into feat/snakemake-orchestration
cailmdaley
committed
ec0617e