Finish the exposure PSF run-dir rename (exp_psf fails on develop) - #888
Open
cailmdaley wants to merge 1 commit into
Open
Finish the exposure PSF run-dir rename (exp_psf fails on develop)#888cailmdaley wants to merge 1 commit into
cailmdaley wants to merge 1 commit into
Conversation
98bc085 renamed the exposure PSF stage's run dir from run_sp_exp_SxSePsfPi to run_sp_exp_SxSePsf, so that config_exp_psfex.ini and config_exp_mccd.ini write one path and nothing downstream of exp_psf has to branch on `psf:`. It updated the one INPUT_DIR that existed at the time and left every other copy of the name behind. The mask_query merge then reintroduced a stale one: the chain grew a module between sextractor and setools, and setools' INPUT_DIR came across from the healsparse branch still spelling the old name. smk-g7 lost all 127 exposures to it, in two different voices: ERROR: Invalid INPUT_DIR (.../run_sp_exp_SxSePsfPi/mask_query_runner/output) on the first attempt -- mask_query had in fact run and written all 40 sexcat_ext files, into run_sp_exp_SxSePsf, where setools was not looking -- and then, on every retry: ERROR: Directory .../run_sp_exp_SxSePsf already exists. which is the SAME bug wearing the STAGE_DIR hat. unit_pre clears the stage's run dir before each attempt precisely so a retry starts clean, and it takes the directory's name from completeness.STAGE_DIR. That entry was stale too, so the rm -rf had been clearing a directory nobody writes and the real one accumulated until ShapePipe refused it. So the retry cleanup needed no new mechanism; it needed the name it already had. Fixing STAGE_DIR fixes both the clearing and the counting. tests/unit/test_workflow_run_names.py holds the invariant that made this possible to get wrong: a run directory is named once in a config and then referred to by things that must find it, and nothing checked that they agreed. It asserts the agreement statically, and fails on the pre-fix tree with both of the errors above named in its messages. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar (cherry picked from commit e9e8564d7aa9b9034c49406bd96852df7ba44525)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hotfix for develop: every
exp_psfjob fails on the committed exposure chain.98bc0857renamed the exposure PSF run dirrun_sp_exp_SxSePsfPi→run_sp_exp_SxSePsfand updated the oneINPUT_DIRthat existed then; themask_querymerge (94c7a863) then brought setools'INPUT_DIRacross from the healsparse branch still spelling the old name. First seen on the smk-g7 campaign (the first workflow run since #847 merged — smk-g6 predates it):ERROR: Invalid INPUT_DIR (…/run_sp_exp_SxSePsfPi/mask_query_runner/output), and on retryDirectory …/run_sp_exp_SxSePsf already exists, becausecompleteness.STAGE_DIR["exp_psf"]— whichunit_pre'srm -rfuses to reset a retried unit — carried the stale name too.Fixes the four consumers (
config_exp_psfex.inisetoolsINPUT_DIR,completeness.STAGE_DIR, two docstrings) and addstests/unit/test_workflow_run_names.py, which asserts a run dir is named once per config and every chainedINPUT_DIR, both PSF configs, andcompleteness.STAGE_DIRagree — it fails on the pre-fix tree naming both production errors. Notecompleteness.pyis hashed into every rule'sparams, so this invalidates in-flight campaigns' rerun state (cheap: prepare usesRETRIEVE=symlink).🤖 Generated with Claude Code
https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar