Classification export: pre-filter 'since' exports by changed alleles in SQL (#1778) - #1779
Open
davmlaw wants to merge 2 commits into
Open
Classification export: pre-filter 'since' exports by changed alleles in SQL (#1778)#1779davmlaw wants to merge 2 commits into
davmlaw wants to merge 2 commits into
Conversation
… in SQL so incremental syncs don't scan every record #1778
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 Written by Claude
Addresses #1778
An incremental (
since=) classification export previously fetched every last-publishedClassificationModification(with all itsselect_relatedjoins andpublished_evidenceJSON), then discarded almost all of them in Python via_passes_since(). A Shariant sync with nothing to download took ~51.5s.Change
ClassificationFilternow computes_since_changed_allele_ids— the set of allele ids with any qualifying change since the date, using four cheap queries that mirror the per-record checks in_passes_since():ClassificationModification.modified >= since(last-published rows)Classification.modified > since_since_flagged_classification_ids)ImportedAlleleInfo.latest_validation.modified > sincecms_qsthen pre-filters to those alleles whensinceis set. Records whoseallele_infohas no matched allele can't be found via the allele-id set, so they're kept in the queryset and left to_passes_since()._passes_since()remains unchanged as the authority — the pre-filter only shrinks what gets fetched and hydrated, and the "whole allele group is exported when any member changed" semantics are preserved (the filter is by allele id, not classification id).When nothing has changed the main query matches zero rows, so a no-op incremental sync should drop from ~50s to well under a second. This benefits all
since-based consumers of the export (Shariant sync, MVL/Alissa downloads).Testing
python3 manage.py test --keepdb classification.tests.utils.test_urlspasses.🤖 Generated with Claude Code