Skip to content

Remap dead Pfam accessions to their #=GF FW successor instead of dropping their DDIs #5

Description

@PelzKo

Summary

Dead Pfam accessions are currently dropped. They could be remapped to their
successor family instead, recovering the DDIs that reference them.

FETCH_DOMAIN_META already reports them correctly — dropped_families.tsv carries
reason dead, and PRUNE_UNREPRESENTED_DDIS removes the DDIs that touch them. So
nothing is wrong today; this is recoverable data being discarded.

The data

Pfam-A.dead.gz (26 KB) carries a forward pointer for every killed family —
measured on the current release: 1121 dead families, 1121 with #=GF FW.

#=GF ID   NtCtMGAM_N
#=GF AC   PF16863
#=GF KL   This family has been killed
#=GF FW   PF13802
#=GF CC   Merged into existing family that has been expanded to cover
#=GF CC   whole structural domain

So PF16863 -> PF13802 is a lookup, not a heuristic.

Dead accessions enter through the DDI sources (3did, PPIDM, Negatome), which are
built against older Pfam releases than the one FETCH_DOMAIN_META streams.

Where the remap has to happen

Downstream, on the DDI rows in the master DB, before EXPORT_UNION_FAMILIES.

Remapping only the instance fetch does not work: the ddi rows would still be keyed
on PF16863 while the instances come back keyed on PF13802, so
PRUNE_UNREPRESENTED_DDIS still deletes them. The rewrite must cover ddi.pfam_a,
ddi.pfam_b and domain.pfam_id.

Implementation notes

  • New param url_pfam_dead (https://ftp.ebi.ac.uk/pub/databases/Pfam/current_release/Pfam-A.dead.gz).
  • New module, between COLLECT_DDI_DATA and EXPORT_UNION_FAMILIES.
  • Collision handling is mandatory: PF16863-X and PF13802-X collapse into one
    DDI once remapped. Needs dedupe on (pfam_a, pfam_b, source, negative), and the
    merge must not silently turn a positive and a negative DDI into one row.
  • Forward pointers can chain (A -> B -> C); resolve transitively with a cycle guard.
  • Report how many DDIs were remapped and how many collapsed, in
    report_ddi_attrition.py — a remap that quietly halves a source is worse than a drop.
  • Semantics caveat worth stating in the docs: #=GF CC usually says "merged into an
    existing family that has been expanded", so the successor family is broader than
    the one the source DDI was annotated against.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions