Fix DBT converts PERCENTILE_DISC to PERCENTILE_CONT on round-trip - #316
Open
MonkeyCanCode wants to merge 2 commits into
Open
Fix DBT converts PERCENTILE_DISC to PERCENTILE_CONT on round-trip#316MonkeyCanCode wants to merge 2 commits into
MonkeyCanCode wants to merge 2 commits into
Conversation
Contributor
Author
|
@QMalcolm mind take a look? |
There was a problem hiding this comment.
🟢 Approval recommended
The fix is small, localized, and includes a targeted regression test that confirms the discrete percentile flag is preserved through the round-trip.
Pull request overview
This PR fixes a DBT converter round-trip bug where PERCENTILE_DISC expressions were parsed and rebuilt as PERCENTILE_CONT, losing the discrete percentile flag during OSSIE ↔ MSI conversions.
Changes:
- Extend
_extract_agg_info()to preserve whether the parsed percentile aggregate is discrete (PERCENTILE_DISC) vs continuous (PERCENTILE_CONT). - Populate
use_discrete_percentileinPydanticMeasureAggregationParameterswhen converting OSSIE → MSI. - Add a round-trip test ensuring
use_discrete_percentile=Truesurvives MSI → OSSIE → MSI.
File summaries
| File | Description |
|---|---|
| converters/dbt/src/ossie_dbt/expression_utils.py | Preserve discrete-vs-continuous percentile info during aggregation parsing. |
| converters/dbt/src/ossie_dbt/ossie_to_msi.py | Thread use_discrete_percentile into MSI aggregation params for percentile metrics. |
| converters/dbt/tests/test_ossie_to_msi.py | Add regression test validating PERCENTILE_DISC round-trips correctly. |
Review details
- Files reviewed: 3/3 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+42
to
45
| Returns ``(agg_type, bare_col, percentile, use_discrete_percentile)`` for recognised patterns, | ||
| ``None`` otherwise. ``percentile`` is only set for ``PERCENTILE`` aggregations; it is ``None`` | ||
| for all others. ``use_discrete_percentile`` is ``True`` only for ``PERCENTILE_DISC``. | ||
| The returned column name has any dataset qualifier stripped. |
Contributor
Author
There was a problem hiding this comment.
Not in scope of this change. I can create a follow up to fix those doc-string later.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Currently
_extract_agg_infofrom https://github.com/apache/ossie/blob/main/converters/dbt/src/ossie_dbt/expression_utils.py#L96 is parse bothexp.PercentileContandexp.PercentileDiscinto the same(agg_type, bare_col, percentile)which dropped the discrete flag. The rebuild later on didn't setuse_discrete_percentile(which default to False) from https://github.com/apache/ossie/blob/main/converters/dbt/src/ossie_dbt/osi_to_msi.py#L287. Thereforce, a later MSI → OSI pass emits PERCENTILE_CONT instead of honoring original PERCENTILE_DISC when in-used.To fix this problem, I added one extra field
use_discrete_percentilein_extract_agg_infoas return value.Here is the local test validation:
Related Issues
Checklist
Specification
core-spec/and follow the existing structureOntology
ontology/are consistent with spec changesConverters
converters/is updated to reflect spec or ontology changesValidation
validation/are updated if the spec changedDocumentation
docs/is updated to reflect any user-facing changesCONTRIBUTING.mdis updated if the contribution process changedExamples
examples/are added or updated for any new spec constructs or converter supportTests
pytest/ CI green)Compliance