Add --keep_msas: refresh templates on existing features without redoing MSAs

[DimaMolod]

Why

AlphaFold3's fetch_databases.sh pins pdb_seqres_2022_09_28.fasta, so local AF3 runs at EMBL had been searching a 2022 snapshot of PDB seqres. The refreshed file is about a third larger. Features generated against the old snapshot need new templates, not new MSAs — but the only way to refresh them was to regenerate everything, redoing hours of jackhmmer/HHblits work to change minutes of hmmsearch output.

What

--keep_msas updates existing features in --output_dir in place, keeping their MSAs and re-running only the template search.

• AlphaFold2 — reloads the monomer, re-runs template search, swaps the template_* block using the same helper TrueMultimer already used for its single-PDB multimeric templates.
• AlphaFold3 — reloads the feature JSON and re-processes it with both MSAs set and templates=None, which is AF3's own "Has MSA, but doesn't have templates. Search for templates only" branch. The distinction matters: templates=[] means "do not search", and a partially populated chain is rejected by AF3 outright, so None is the specific signal required.

Proteins with no stored features are generated normally, so the flag is safe to apply to a whole sample sheet. It takes precedence over --skip_existing, and is rejected in combination with --use_mmseqs2 (MSAs and templates arrive together, with no separable search) or --skip_msa (no MSAs to keep).

One caveat worth knowing

DataPipeline.process builds the template profile from the uniref90 alignment alone, not the merged MSA that lands in the features. So uniref90_hits.sto is preferred when present. When --save_msa_files=False deleted it, the alignment is rebuilt from the stored integer MSA — which is the merged alignment, giving a deeper profile that can return different hits. That is defensible but is not a reproduction of a fresh run, so the source is logged and recorded on the monomer as template_msa_source rather than passing silently.

Two related fixes found in the same round

Validate precomputed MSAs before reuse. --use_precomputed_msas trusts whatever it finds, so an alignment left behind by a killed job is read back as if complete. An empty uniref90_hits.sto surfaces as StopIteration inside deduplicate_stockholm_msa, naming neither the file nor the protein; a partially written one silently produces shallower features, which is worse because it is indistinguishable from a good result. Unsound files are now removed so they regenerate. Search results (pdb_hits.*) are exempt — an empty one just means no hits were found.

Follow symlinks when recording database provenance. Sites that refresh a database in place keep AF3's pinned name as a symlink to the current file, and the release date is parsed out of the file name. Metadata therefore reported 2022-09-28 while 2026-08-19 data was actually searched — and that date ends up in methods sections. Both ends of the symlink are now recorded.

Verification

End to end against the real databases on an existing 74 aa feature set, in both directions, with hmmsearch querying the live pdb_seqres:

--max_template_date	templates	MSA
1990-01-01	6 → none	byte-identical (11, 74)
2050-01-01	6 → 10 (5 kept, 1 dropped, 5 new)	byte-identical (11, 74)

Deletion matrices identical in both. The reshuffle in the second run is the reconstruction caveat above — that feature set had no saved uniref90_hits.sto.

62 new tests across test/unit and test/integration; 496 pass in total. The AF3 half runs against the stubbed folding_input the suite already uses, so it stays covered where AF3 is not built.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 52552bf8db

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Validate AF3 cached sequence before reusing MSAs

When --keep_msas finds an existing AF3 JSON, this path reuses it based only on the filename/description and never checks that the cached chain sequence matches the current FASTA sequence. If a target is edited but keeps the same description, the pipeline will re-run template search using the old chain and old MSAs, then overwrite the feature file with features for the wrong protein instead of falling back to regeneration as the AF2 path does on sequence mismatch.

Useful? React with 👍 / 👎.