Summary
Dead Pfam accessions are currently dropped. They could be remapped to their
successor family instead, recovering the DDIs that reference them.
FETCH_DOMAIN_META already reports them correctly — dropped_families.tsv carries
reason dead, and PRUNE_UNREPRESENTED_DDIS removes the DDIs that touch them. So
nothing is wrong today; this is recoverable data being discarded.
The data
Pfam-A.dead.gz (26 KB) carries a forward pointer for every killed family —
measured on the current release: 1121 dead families, 1121 with #=GF FW.
#=GF ID NtCtMGAM_N
#=GF AC PF16863
#=GF KL This family has been killed
#=GF FW PF13802
#=GF CC Merged into existing family that has been expanded to cover
#=GF CC whole structural domain
So PF16863 -> PF13802 is a lookup, not a heuristic.
Dead accessions enter through the DDI sources (3did, PPIDM, Negatome), which are
built against older Pfam releases than the one FETCH_DOMAIN_META streams.
Where the remap has to happen
Downstream, on the DDI rows in the master DB, before EXPORT_UNION_FAMILIES.
Remapping only the instance fetch does not work: the ddi rows would still be keyed
on PF16863 while the instances come back keyed on PF13802, so
PRUNE_UNREPRESENTED_DDIS still deletes them. The rewrite must cover ddi.pfam_a,
ddi.pfam_b and domain.pfam_id.
Implementation notes
- New param
url_pfam_dead (https://ftp.ebi.ac.uk/pub/databases/Pfam/current_release/Pfam-A.dead.gz).
- New module, between
COLLECT_DDI_DATA and EXPORT_UNION_FAMILIES.
- Collision handling is mandatory:
PF16863-X and PF13802-X collapse into one
DDI once remapped. Needs dedupe on (pfam_a, pfam_b, source, negative), and the
merge must not silently turn a positive and a negative DDI into one row.
- Forward pointers can chain (
A -> B -> C); resolve transitively with a cycle guard.
- Report how many DDIs were remapped and how many collapsed, in
report_ddi_attrition.py — a remap that quietly halves a source is worse than a drop.
- Semantics caveat worth stating in the docs:
#=GF CC usually says "merged into an
existing family that has been expanded", so the successor family is broader than
the one the source DDI was annotated against.
Summary
Dead Pfam accessions are currently dropped. They could be remapped to their
successor family instead, recovering the DDIs that reference them.
FETCH_DOMAIN_METAalready reports them correctly —dropped_families.tsvcarriesreason
dead, andPRUNE_UNREPRESENTED_DDISremoves the DDIs that touch them. Sonothing is wrong today; this is recoverable data being discarded.
The data
Pfam-A.dead.gz(26 KB) carries a forward pointer for every killed family —measured on the current release: 1121 dead families, 1121 with
#=GF FW.So
PF16863 -> PF13802is a lookup, not a heuristic.Dead accessions enter through the DDI sources (3did, PPIDM, Negatome), which are
built against older Pfam releases than the one
FETCH_DOMAIN_METAstreams.Where the remap has to happen
Downstream, on the DDI rows in the master DB, before
EXPORT_UNION_FAMILIES.Remapping only the instance fetch does not work: the
ddirows would still be keyedon
PF16863while the instances come back keyed onPF13802, soPRUNE_UNREPRESENTED_DDISstill deletes them. The rewrite must coverddi.pfam_a,ddi.pfam_banddomain.pfam_id.Implementation notes
url_pfam_dead(https://ftp.ebi.ac.uk/pub/databases/Pfam/current_release/Pfam-A.dead.gz).COLLECT_DDI_DATAandEXPORT_UNION_FAMILIES.PF16863-XandPF13802-Xcollapse into oneDDI once remapped. Needs dedupe on
(pfam_a, pfam_b, source, negative), and themerge must not silently turn a positive and a negative DDI into one row.
A -> B -> C); resolve transitively with a cycle guard.report_ddi_attrition.py— a remap that quietly halves a source is worse than a drop.#=GF CCusually says "merged into anexisting family that has been expanded", so the successor family is broader than
the one the source DDI was annotated against.