Include the 2026 reporting year in the pipeline - #21
Open
zuhdil wants to merge 5 commits into
Open
Conversation
The column holds Dataverse's rolling "current R&D stage" value, not a frozen 2025 archive. The old name implied the latter, which is why the 2026 rollover went unnoticed when IGH froze the pipeline-inclusion column but not the stage one. Pure rename with no behavioural change. The comment added to `_rdstage_cols` records why a rolling field is pinned to the 2025 boundary and forward-fills into 2026.
IGH closed the 2025 data collection and opened 2026, freezing 2025 into `new_includeinpipeline2025` and rolling the unsuffixed `new_includeinpipeline` forward. Both places that read the unsuffixed column were therefore reporting 2026 values as 2025: the temporal boundary table and the strict-2025 flag behind the WHO Priority page. The visible symptom was the 2025 bucket carrying 8826 rows against 8582 for every prior year — the 244 candidates Dataverse created during 2026, backdated into a year they did not exist in. The R&D stage series is deliberately left pinned at 2025. It has no frozen archive column, and only 40 of 9388 candidates have had their stage touched since the rollover, so recovering a 2025 stage snapshot from an older Bronze backup would buy 0.4% accuracy in exchange for a permanent second ETL input. The existing cross-group forward-fill carries the rolling stage into 2026 on its own. The base test fixture keeps both the frozen and rolling columns populated. Replacing one with the other would leave the new 2026 boundary with no test coverage at all.
Covers both halves of the fix at the star-schema level: that 2026 appears in `fact_pipeline_snapshot`, and that the 2025 and 2026 boundaries read distinct source columns rather than both drawing from the rolling field. Deliberately does not assert row-count parity between 2025 and 2024. All candidates Dataverse creates in 2026 carry a rolling R&D stage, and the stage series has no frozen 2025 archive column of its own, so each such candidate still contributes a stage-only 2025 row. That row carries no pipeline-inclusion value, and every portal query filters on inclusion, so it stays invisible everywhere that matters. Moving the stage series to a 2026 boundary was considered and rejected: it would strip or change 2025 stage data for thousands of existing candidates just to remove a few hundred rows nothing reads. The unit test makes that trade-off explicit at the source: a 2026- created candidate with a stage gets a 2025 boundary with NULL inclusion, alongside its real 2026 boundary with the inclusion value.
Collapses a three-line chained call onto one line in test_candidate_created_in_2026_gets_stage_only_2025_row. Pure whitespace change; no assertion or docstring touched.
The branch fixed the code that mislabelled 2026 data as 2025, but four comments and docstrings still told the old story: - A test docstring claimed 2026-created candidates "must not be backdated into 2025" and blamed a 244-row inflation defect. That prediction was retracted during review: 2025 stayed at 8826 rows, because those candidates carry a rolling R&D stage that is deliberately pinned at 2025, so they correctly get a 2025 row. The docstring now describes what the test actually exercises — the no-stage case, where only the rolling pipeline column contributes a boundary — and points to the adjacent test for the representative with-stage case. - The strict-2025 pipeline-inclusion flag's comment in schema_map.py still named the pre-rollover source column `new_includeinpipeline`; it reads `new_includeinpipeline2025` now that Dataverse rolled its "current" column forward a year. - An e2e docstring said stage-only 2025 rows carry `include_in_pipeline` NULL. The raw `includeinpipeline` column is NULL on those rows, but step 7 of `transform_candidates` maps NaN to 0, so the derived column in Silver and Gold is 0, not NULL. The conclusion (invisible to every `include_in_pipeline = 1` filter) is unchanged; only the mechanism was misdescribed. - Two comments in test_includeinpipeline_2025_raw_preserved_at_candidate_grain said "bronze new_includeinpipeline"; the test reads the frozen new_includeinpipeline2025 column. The fixture happens to give both columns identical values, so this test can't distinguish them — the real frozen-vs-rolling pinning is asserted by test_includeinpipeline_2025_raw_reads_frozen_not_rolling — but the comments should still name the column the test actually reads. No executable code, assertions, or fixture values change in this commit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
Dataverse archives each closed reporting year into a year-suffixed column and keeps one unsuffixed "current" column that rolls forward. Between the July and August bronze snapshots IGH performed that rollover: 2025 was frozen into
new_includeinpipeline2025, and the unsuffixednew_includeinpipelinemoved on to carry 2026.Two places still read the unsuffixed column, so 2026 data was not excluded from the portal — it was mislabelled as 2025:
_expand_temporal_rows, pinned to2025-01-01.new_includeinpipelinewas Yes in 2025".This was already live. The deployed star schema had 8826 rows in 2025 against 8582 for every prior year — the 244 candidates Dataverse created during 2026, backdated into a year they did not exist in.
Nothing ever actually blocked 2026. There is no year filter in
igh-data-sync, and the dashboard derives its year list from the data. The hardcoded boundary table was the only gate.The change
new_includeinpipeline2025; the rolling column gets a new 2026 boundary._resolved_rdstage_2025renamed to_resolved_rdstage_current— the old name implied a frozen archive and is why this drifted unnoticed.The R&D stage series is deliberately left pinned at 2025
It has no frozen archive column;
_vin_currentrndstage_valueis a single rolling field. The cross-group forward-fill carries it into 2026 on its own.Moving it was measured against the bronze DB and rejected: only 40 of 9388 candidates have had their stage touched since the rollover, while moving the boundary would cost 3998 candidates their 2025 stage entirely and change 1104 more.
One consequence, documented in the spec: all 244 candidates created in 2026 carry a rolling stage, so each keeps a stage-only 2025 row. Those rows derive
include_in_pipeline = 0and are invisible to every portal query, all of which filter on it. An earlier draft predicted 2025 would fall to 8582 rows; that was wrong and has been corrected.Verification
fact_pipeline_snapshotbash scripts/qa.shFive new unit tests plus two e2e assertions. The e2e divergence test is a real regression guard: repointing 2025 back to the rolling column makes forward-fill produce identical counts, and it fails.
Deferred
When IGH freezes 2026 and rolls to 2027, both new e2e tests still pass while 2027 lands silently in the 2026 bucket — the same failure mode. No alarm was built, since that is new scope, but it is recorded in the spec and worth doing before the 2027 collection opens.
Related
Requires akvo/igh-dashboard#174, which refreshes the star schema and re-baselines the counts this change moves.
🤖 Generated with Claude Code