fix: preserve_in_zip never replaces an existing archive member

[Jammy2211]

Overview

Paths.preserve_in_zip (autofit/non_linear/paths/abstract.py:316) adds a file to a completed search's
.zip only when that member is absent:

with zipfile.ZipFile(self._zip_path, "a") as f:
    if arcname not in f.namelist():
        f.write(file_path, arcname)

When the member already exists but its on-disk counterpart has been rewritten, the archive keeps the old
bytes. Because restore() deletes the output directory and re-extracts the zip, the rewritten file is
discarded on the next resume and the stale member comes back.

How it was found

Spun out of PyAutoGalaxy#516. The per-galaxy adapt-image cache (files/galaxy_images_snr.fits) can go stale
when a rerun's dataset mask changes while the model stays identical — the search identifier does not encode
the dataset, so the run lands on the previous run's output directory. PyAutoGalaxy#516 fixes the correctness
half: the cached mask is validated on load and a mismatch becomes a cache miss, so the images are recomputed
on the current mask.

The recomputed images are then written back via _galaxy_image_dict_to_cache -> preserve_in_zip. Verified
end-to-end that the archived copy is not updated: after a run that successfully invalidated and
recomputed four stale caches, three upstream searches' zips still held the 15x15 stale member (only the one
search whose zip was newly created held the correct 16x16 member).

Consequence is a performance wart, not a correctness one: such a search misses its cache on every run
rather than once, paying the max-log-likelihood-fit rebuild each time, until its output directory is cleared.

Plan

• Make preserve_in_zip replace an existing member whose content differs, instead of skipping it. zipfile
  cannot overwrite a member in place, so the archive needs rewriting when a replacement is required — keep
  the no-op fast path for the already-identical case so the common path does not pay a rewrite.
• Preserve the existing no-op when the zip does not yet exist (the search is still running and everything is
  zipped at completion).
• Add a unit test: write a member, rewrite the on-disk file, preserve_in_zip again, assert the archive
  carries the new bytes and that restore() yields them.

Notes

• Not urgent for the PyAutoGalaxy#516 fix, which is correct without it — this only restores the caching win.
• Worth checking the other preserve_in_zip callers for the same silently-stale assumption.

[Jammy2211]

Plan (approved 2026-07-28) — feature/preserve-in-zip-replace-member

Mind prompt: bug/autofit/preserve_in_zip_replace_stale_member.md · classification library (PyAutoFit + a docs-only PyAutoGalaxy follow-up).

High level

• Make AbstractPaths.preserve_in_zip replace an existing member whose bytes differ from the on-disk file, instead of silently skipping it.
• Keep both existing exits untouched: no zip → no-op; member absent → plain append.
• Detect "already identical" cheaply (size + CRC from the central directory) so the common re-preserve of an unchanged cache never pays an archive rewrite.
• Rewrite the archive atomically when a replacement is genuinely needed, since zipfile cannot overwrite a member in place.
• Unit-test both new paths: stale member is refreshed and survives restore(); identical content leaves the archive byte-identical.
• Follow up in PyAutoGalaxy with a docstring correction — galaxy_name_image_dict_via_result_from currently documents the un-repairable stale cache as standing behaviour.

Detailed plan

Work classification: Library · Worktree root: ~/Code/PyAutoLabs-wt/preserve-in-zip-replace-member/

1. autofit/non_linear/paths/abstract.py — preserve_in_zip

Three-way behaviour:

• zip missing → return (unchanged; the running search zips everything at completion).
• arcname absent → append via ZipFile(..., "a") (unchanged fast path).
• arcname present → compare the archived member with the on-disk file using the ZipInfo already in the central directory: info.file_size vs os.path.getsize, and info.CRC vs a chunk-streamed zlib.crc32 (so a large FITS cache is never read fully into memory).
  • identical → no-op, archive untouched.
  • different → rewrite.

Private module-level rewrite helper: stream every member except arcname from the old zip into a temp zip in the same directory (so the final os.replace is atomic on one filesystem), preserving each member's ZipInfo (date_time, compress_type, external attrs) via dest.open(zinfo, "w") + shutil.copyfileobj; write the replacement under arcname with the old member's compress_type (fallback ZIP_DEFLATED, matching zip_directory in autofit/tools/util.py); os.replace(tmp, zip_path). Unlink the temp and re-raise on any exception — the original archive is never left half-written.

Docstring: "skip if the member is already present" → "replace an existing member whose content differs; no-op when byte-identical".

2. test_autofit/non_linear/paths/test_paths.py

Existing test__preserve_in_zip__file_survives_restore / __no_zip_is_a_no_op stay. Add:

• test__preserve_in_zip__replaces_stale_member — preserve, rewrite the on-disk file, preserve again; assert the archive holds the new bytes, the member appears exactly once, the unrelated files/samples_summary.json member survives, and restore() yields the new bytes.
• test__preserve_in_zip__identical_content_does_not_rewrite — preserve twice unchanged; assert the zip's raw bytes are byte-identical, pinning the no-rewrite fast path.

3. PyAutoGalaxy follow-up (docs-only, merges after the Fit PR)

autogalaxy/analysis/adapt_images/adapt_images.py → galaxy_name_image_dict_via_result_from docstring: the paragraph stating a stale cache "is not repaired on disk … misses the cache on every run … (PyAutoFit#1414)" becomes the repaired behaviour — the recompute is written back and the archived member is refreshed, so a mask change costs one recompute, not one per run.

Other callers checked (issue's Notes item): autolens/analysis/result.py:316 writes its multiple-image-positions cache only when the loose file is absent, so it never depended on replacement — no change needed.

Verification

pytest test_autofit/non_linear/paths/test_paths.py, then the full test_autofit suite (preserve_in_zip/restore sit on the resume path); a hand round-trip in the worktree (zip → preserve → mutate → preserve → restore()); PyAutoGalaxy test_autogalaxy/analysis/test_adapt_images.py for the docs-only change.

Scoping note

The Brain bug agent returned large / split-into-phases, scored off the two-repo count. The actual change is one method + two tests + one docstring paragraph, so the phase split is overridden: one PyAutoFit PR, one PyAutoGalaxy docs PR.

[Jammy2211]

Smoke Test Results — 2026-07-28

Run on feature/preserve-in-zip-replace-member (PyAutoFit + PyAutoGalaxy from the worktree, remaining libraries from main).

Workspace	Passed	Failed	Skipped	Total
autofit_workspace	7	0	1	8
autogalaxy_workspace	10	0	0	10
autolens_workspace	14	0	0	14
autolens_workspace_test	13	6	2	21
euclid_strong_lens_modeling_pipeline	6	0	0	6
HowToLens	6	0	0	6

All 6 failures are pre-existing on main, not caused by this branch — they are part of the Heart YELLOW reason "workspace validation not passing (13 failed, 2026-07-21)".

Failures + baselines

Five are JAX-vs-numpy likelihood parity mismatches in autolens_workspace_test/*/jax_likelihood/. Each was re-run sequentially against unmodified main libraries and reproduced with byte-identical numbers:

Script	Branch	main baseline
imaging/jax_likelihood/lp.py	max rel diff 0.0821195	identical
imaging/jax_likelihood/rectangular.py	max rel diff 0.92252463	identical
imaging/jax_likelihood/mge.py	max rel diff 0.94615314	identical
multi/jax_likelihood/mge.py	max rel diff 0.97171747	identical
interferometer/jax_likelihood/rectangular.py	max rel diff 0.00012399	identical

The sixth, imaging/subhalo_recovery.py, hit the harness's 300s per-script cap (exit 124). Re-run sequentially and unloaded against main, it is still running past 8 minutes — i.e. it exceeds the cap on main too, and is not marked # SLOW in config/build/no_run.yaml. Unrelated to this change (a fresh run never reaches the replacement path, which only fires when an archive member already exists and its content differs).

Library gates

• PyAutoFit pytest test_autofit: 1559 passed, 1 skipped
• PyAutoGalaxy pytest test_autogalaxy: 1009 passed
• PR CI PyAutoFit#1427: unittest 3.12 + 3.13 + docs green
• Heart: YELLOW (4 reasons, none touching this change) — acknowledged by the human at ship time

[Jammy2211]

Baseline follow-up: imaging/subhalo_recovery.py passes on unmodified main — exit 0 in 8m25s, sequential and unloaded. So it is not broken, just ~505s past the harness's 300s per-script cap, and it carries no # SLOW marker. Filed separately; nothing to do with this change.

[Jammy2211]

Shipped

• fix: preserve_in_zip replaces a stale archive member (#1414) #1427 — MERGED (preserve_in_zip replaces a stale member; 2 new unit tests)
• docs: adapt-image cache is repaired on disk (PyAutoFit#1414) PyAutoGalaxy#533 — MERGED (docstring: the adapt-image cache is repaired on disk now)

preserve_in_zip keeps its two existing exits (no zip → no-op; member absent → append) and its no-op when the archived member is byte-identical, so the common resume path never pays a rewrite. Only a genuinely changed member triggers the atomic archive rewrite.

Closing.