Add strict 2025 pipeline-inclusion flag to dim_candidate_core - #17
Merged
Conversation
The WHO Priority page must count only candidates whose 2025 `new_includeinpipeline` is exactly Yes. The existing `includeinpipeline` flag forward-fills from earlier years during temporal expansion, so a blank 2025 value can still read as included. Capture the raw 2025 value into a candidate-grain `includeinpipeline_2025_raw` column before `_expand_temporal_rows` consumes and drops `new_includeinpipeline`, so the gold layer can derive a strict 2025-only flag.
drop_empty_columns would otherwise silently remove the new column when every candidate's 2025 inclusion is blank (or the source column is absent), leaving the gold layer to KeyError on a missing column. Add it to the preserve list so the schema stays consistent, and cover the all-null case with a test. Tighten the existing grain test's `len(rows_a) >= 1` to `> 1` to confirm the value fans out across all temporal rows for candidate id-1, not just one.
Map the captured `includeinpipeline_2025_raw` to a new `new_include_in_pipeline_2025` boolean that is 1 only when the 2025 value is exactly Yes. This is deliberately separate from the forward-filled `include_in_pipeline` fact flag so existing pages keep their behaviour while the WHO Priority page can filter strictly.
…core
Adds an e2e assertion that `new_include_in_pipeline_2025` exists in
`dim_candidate_core` and holds only integer 0/1 values after the full
bronze→silver→gold transform.
Running the test against the real bronze DB revealed that the column was
stored as TEXT ('0'/'1') rather than INTEGER. The DDL generator infers
column affinity by name suffix, but `new_include_in_pipeline_2025` does
not end in any of the known INTEGER suffixes (_key, _id, _flag, _count),
so it defaulted to TEXT. Adding it to `INTEGER_EXACT_NAMES` in
`ddl_generator.py` fixes the affinity, which also ensures the column
reads back as integers wherever the gold schema is queried in production.
3 tasks
CI runs `ruff format --check src/ tests/`; the long assert line in the new dim_candidate_core strict-flag test exceeded the line length. Wrap it to match the formatter.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a strict 2025 pipeline-inclusion flag to the candidate dimension, for the WHO Priority page's drill-down.
new_includeinpipelinecolumn is the 2025 source-of-truth, but the existing goldfact_pipeline_snapshot.include_in_pipelineflag is forward-filled during temporal expansion — so a blank 2025 value can still read as "included". (34 candidates currently fall in this gap.)includeinpipeline_2025_rawbefore_expand_temporal_rowsconsumes and dropsnew_includeinpipeline(and protect it throughdrop_empty_columnswhen all-null).dim_candidate_core.new_include_in_pipeline_2025=CASE WHEN includeinpipeline_2025_raw = 100000000 THEN 1 ELSE 0 END— Yes-only; No, Pending and blank all map to 0.INTEGER_EXACT_NAMESso it gets INTEGER (not TEXT) affinity, since its name matches no INTEGER suffix.The existing
include_in_pipelineflag is left untouched, so every other page keeps its current behaviour.Coordination
Paired with akvo/igh-dashboard#148 (the WHO page consumes this column). That PR bundles a rebuilt
star_schema.dbso it is self-contained for CI, but this ETL change is the source-of-truth for how the column is produced — merge this so future rebuilds carry it.Test Plan
uv run pytest— 442 passed (unit incl. new candidate-capture + schema-map tests)uv run pytest --e2e -k silver_to_gold— gold column present and strictly 0/1new_includeinpipeline = Yes, zero leak