Skip to content

Classification export: pre-filter 'since' exports by changed alleles in SQL (#1778) - #1779

Open
davmlaw wants to merge 2 commits into
masterfrom
export_since_prefilter
Open

Classification export: pre-filter 'since' exports by changed alleles in SQL (#1778)#1779
davmlaw wants to merge 2 commits into
masterfrom
export_since_prefilter

Conversation

@davmlaw

@davmlaw davmlaw commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

🤖 Written by Claude

Addresses #1778

An incremental (since=) classification export previously fetched every last-published ClassificationModification (with all its select_related joins and published_evidence JSON), then discarded almost all of them in Python via _passes_since(). A Shariant sync with nothing to download took ~51.5s.

Change

ClassificationFilter now computes _since_changed_allele_ids — the set of allele ids with any qualifying change since the date, using four cheap queries that mirror the per-record checks in _passes_since():

  • ClassificationModification.modified >= since (last-published rows)
  • Classification.modified > since
  • classifications flagged since (reuses _since_flagged_classification_ids)
  • ImportedAlleleInfo.latest_validation.modified > since

cms_qs then pre-filters to those alleles when since is set. Records whose allele_info has no matched allele can't be found via the allele-id set, so they're kept in the queryset and left to _passes_since().

_passes_since() remains unchanged as the authority — the pre-filter only shrinks what gets fetched and hydrated, and the "whole allele group is exported when any member changed" semantics are preserved (the filter is by allele id, not classification id).

When nothing has changed the main query matches zero rows, so a no-op incremental sync should drop from ~50s to well under a second. This benefits all since-based consumers of the export (Shariant sync, MVL/Alissa downloads).

Testing

  • python3 manage.py test --keepdb classification.tests.utils.test_urls passes.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant