instance_tier: implement all_species_any_review_status (full TrEMBL universe)

[PelzKo]

Summary

instance_tier will gain four explicit values:

value	strata filled	status
human_reviewed	human Swiss-Prot	implemented
human_any_review_status	+ human TrEMBL	being implemented
all_species_reviewed	+ non-human Swiss-Prot	being implemented
all_species_any_review_status	+ all TrEMBL	not implemented — this issue

The first three fit the current architecture. The fourth does not, and the reason is
worth writing down so the next attempt does not start by trying it the easy way.

Why the fourth does not fit

ppi-splitting-pipeline's bin/fetch_domains.py builds the protein universe by
parsing a UniProt flat file fully into RAM:

universe = parse_uniprot_dat(iter_gzip_lines(fh), taxa_filter)   # {acc: (taxon, seq)}

then streams Pfam-A.regions.tsv.gz and looks each region's parent up in that dict.

Sizes (UniProt release 2026_03):

file	size (gz)	entries
uniprot_sprot.dat.gz	667 M	~573 k
uniprot_trembl.dat.gz	110 G	~250 M

all_species_reviewed holds ~573 k proteins — roughly 1 GB of Python strings, which
works. all_species_any_review_status would hold ~250 M, which is hundreds of GB. It
is not a tuning problem.

content_digest() is a second, independent wall: it sorts and joins every accession
in the universe to build the cache key (0.3 s at 573 k). At 250 M that is minutes of
CPU and tens of GB of transient string.

What implementing it actually requires

Not a bigger machine — a different join strategy. Sketch:

1. Build a cached accession index over the full TrEMBL DAT, once per UniProt
   release: accession -> (taxon_id, reviewed, byte_offset). ~250 M rows; on disk
   (SQLite / LMDB / sorted binary), not in RAM. This is the expensive step (~110 GB
   streamed once) and it is exactly what the interpro_cache is for.
2. Tier-assign during the regions pass using that index, so stratification stays
   correct — a family that has a human member must still get it, which a
   "sample first, resolve taxon later" shortcut cannot guarantee.
3. Fetch sequences only for the sampled instances, by seeking to the recorded
   offsets. Sampled set is bounded by ddi_examples_target * ddi_examples_pool_factor * n_families (currently 25 x ~20 k = ~500 k), so this stays small.
4. Replace content_digest() with an order-independent accumulator (XOR/sum of
   per-accession hashes) or with a release-string + file-size digest.
5. Bump FORMAT_VERSION again.

Disk budget is not the constraint (cache is provisioned at 1 TB; 110 GB TrEMBL +
4.7 GB regions + a ~150 GB STRING working set still fits).

Where the code lives

• Instance sampling and the universe: bin/fetch_domains.py in
  bionetslab/ppi-splitting-pipeline (vendored here as
  subworkflows/external/ppi-splitting, read-only).
• The instance_tier / uniprot_dat_urls parameter surface, bin/parse_swissprot_dat.py
  and the workflow guards: this repo.

Both halves are needed: parse_swissprot_dat.py also parses the flat files, to fill
protein.sequence, protein_go_terms and the STRING id map, and has the same
streaming problem at full-TrEMBL scale.