Compute a canonical disease display label in the ETL - #16
Merged
Conversation
The portal needs to print one disease name consistently across the
Pipeline Explorer tables, the slide-ins, and the per-column table
filter. Compute that label once here, beside the existing secondary-
sentinel and STI-collapse normalisation, and carry it into Gold as a
new `dim_disease.disease_label` column. The rule:
* default -> the secondary disease (COVID-19, Lassa fever,
Gonorrhea);
* no secondary -> the primary disease group (Tuberculosis, Buruli
ulcer);
* Malaria -> "<primary> – <secondary>" with a spaced en dash,
because its strains (P. falciparum, P. vivax)
never stand alone.
STIs need no special branch: they arrive parent-collapsed with the
specific infection already in `secondary_disease_name`, so the default
branch prints it. Computing the label in one tested place means the
downstream GraphQL backend and React tables/slide-ins are pure pass-
throughs with no duplicated rule.
The computation is guarded on both columns being present, matching the
existing STI-collapse guard: in production both always carry values, so
the column is always materialised.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Compute a single canonical disease display label in the ETL and carry it
into Gold as a new
dim_disease.disease_labelcolumn. This is the data-layerhalf of the client's final disease-name rule; the dashboard reads this column
(separate PR in
igh-dashboard).The rule
With
P = disease_filterandS = secondary_disease_name(the"No secondary disease"sentinel and blanks are already normalised to NULLupstream):
P + " – " + S(e.g.Malaria – P. falciparum) — its strainsnever stand alone, so the primary is prefixed with a spaced en dash.
S(COVID-19, Lassa fever, Gonorrhea).P(Tuberculosis, Buruli ulcer).STIs need no special branch: they arrive parent-collapsed with the specific
infection in
secondary_disease_name, so the default branch prints it.Why compute it here
Computing the label once in
diseases.py, beside the existing secondary-sentineland STI normalisation (and unit-tested alongside them), keeps the rule in a single
tested place. The downstream GraphQL backend and React tables/slide-ins become
pure pass-throughs with no duplicated logic.
Changes
transformations/diseases.py— computedisease_labelafter the existingdisease-filter normalisation.
silver_to_gold/config/schema_map.py— passdisease_labelthrough todim_disease.tests/unit/test_diseases.py— cases for the default, no-secondary, Malaria,STI, and both-null branches.
tests/e2e/test_silver_to_gold_e2e.py— adddisease_labelto the expecteddim_diseasecolumn set.Verification
uv run pytest tests/unit/test_diseases.py— 22 pass.Malaria – <strain>,secondary-bearing rows render the secondary, no-secondary rows render the
primary group, 239 cross-cutting rows are NULL.