_save_ngmix_data pre-fills every NGMIX_* column with sentinels and overwrites them only for objects found in the ngmix output catalogue. An object ngmix never fit is absent from that catalogue (it raised "0 epoch to process", ngmix.py:2034, or any other exception caught at ngmix.py:1058-1063, and was skipped with continue), so it keeps every default:
NGMIX_G1/G2 = -10 make_cat.py:494
NGMIX_T, NGMIX_FLUX = 0 make_cat.py:474
NGMIX_MCAL_FLAGS = 0 make_cat.py:501
NGMIX_N_EPOCH = 0 make_cat.py:453
MCAL_FLAGS, the column meaning "the metacal fit failed", therefore reads clean for objects that were never fit at all. In final_cat_smk-g7.hdf5 (the 64-tile acceptance run of #879/#886/#887/#880) this is 18,983 of 1,851,100 objects (1.03%): cutting on MCAL_FLAGS == 0 alone gives mean e1 = −0.107 (std 1.03), against −0.004 (std 0.22) once NGMIX_N_EPOCH > 0 is also required.
Minimal fix: initialise NGMIX_MCAL_FLAGS at make_cat.py:501 to a nonzero "not fitted" value instead of 0, so the flag column is honest by default and only a real ngmix row can clear it. A dedicated bit would be preferable to a bare nonzero if the column gains a documented bit table. sp_validation's classification_galaxy_ngmix happens to reject these rows today through its NGMIX_G1_PSF_ORIG_NOSHEAR != -10 guard — an exact float comparison against a sentinel — and CosmoStat/sp_validation#343 now carries an explicit NGMIX_N_EPOCH > 0 guard instead.
— Claude (Fable) on behalf of Cail
_save_ngmix_datapre-fills everyNGMIX_*column with sentinels and overwrites them only for objects found in the ngmix output catalogue. An object ngmix never fit is absent from that catalogue (it raised "0 epoch to process",ngmix.py:2034, or any other exception caught atngmix.py:1058-1063, and was skipped withcontinue), so it keeps every default:MCAL_FLAGS, the column meaning "the metacal fit failed", therefore reads clean for objects that were never fit at all. Infinal_cat_smk-g7.hdf5(the 64-tile acceptance run of #879/#886/#887/#880) this is 18,983 of 1,851,100 objects (1.03%): cutting onMCAL_FLAGS == 0alone gives mean e1 = −0.107 (std 1.03), against −0.004 (std 0.22) onceNGMIX_N_EPOCH > 0is also required.Minimal fix: initialise
NGMIX_MCAL_FLAGSatmake_cat.py:501to a nonzero "not fitted" value instead of 0, so the flag column is honest by default and only a real ngmix row can clear it. A dedicated bit would be preferable to a bare nonzero if the column gains a documented bit table. sp_validation'sclassification_galaxy_ngmixhappens to reject these rows today through itsNGMIX_G1_PSF_ORIG_NOSHEAR != -10guard — an exact float comparison against a sentinel — and CosmoStat/sp_validation#343 now carries an explicitNGMIX_N_EPOCH > 0guard instead.— Claude (Fable) on behalf of Cail