Skip to content

Fix DBT converts PERCENTILE_DISC to PERCENTILE_CONT on round-trip - #316

Open
MonkeyCanCode wants to merge 2 commits into
apache:mainfrom
MonkeyCanCode:dbt_from_PERCENTILE_DISC_2_PERCENTILE_CONT
Open

Fix DBT converts PERCENTILE_DISC to PERCENTILE_CONT on round-trip#316
MonkeyCanCode wants to merge 2 commits into
apache:mainfrom
MonkeyCanCode:dbt_from_PERCENTILE_DISC_2_PERCENTILE_CONT

Conversation

@MonkeyCanCode

Copy link
Copy Markdown
Contributor

Summary

Currently _extract_agg_info from https://github.com/apache/ossie/blob/main/converters/dbt/src/ossie_dbt/expression_utils.py#L96 is parse both exp.PercentileCont and exp.PercentileDisc into the same (agg_type, bare_col, percentile) which dropped the discrete flag. The rebuild later on didn't set use_discrete_percentile (which default to False) from https://github.com/apache/ossie/blob/main/converters/dbt/src/ossie_dbt/osi_to_msi.py#L287. Thereforce, a later MSI → OSI pass emits PERCENTILE_CONT instead of honoring original PERCENTILE_DISC when in-used.

To fix this problem, I added one extra field use_discrete_percentile in _extract_agg_info as return value.

Here is the local test validation:

➜  dbt git:(dbt_from_PERCENTILE_DISC_2_PERCENTILE_CONT) ✗ uv run pytest
==================================================================================================================================================== test session starts =====================================================================================================================================================
platform darwin -- Python 3.11.13, pytest-9.1.1, pluggy-1.6.0
rootdir: /Users/yong/Desktop/GitHome/ossie/converters/dbt
configfile: pyproject.toml
testpaths: tests
plugins: syrupy-5.5.3
collected 100 items

tests/test_msi_to_osi.py ..................................................................                                                                                                                                                                                                                            [ 66%]
tests/test_osi_to_msi.py ..................................                                                                                                                                                                                                                                                            [100%]

-------------------------------------------------------------------------------------------------------------------------------------------------- snapshot report summary ---------------------------------------------------------------------------------------------------------------------------------------------------
5 snapshots passed.
==================================================================================================================================================== 100 passed in 0.36s =====================================================================================================================================================

Related Issues

Checklist

Specification

  • Spec changes are included in core-spec/ and follow the existing structure
  • Spec changes have been discussed on the mailing list or in a linked issue
  • Breaking changes to the spec are clearly called out in the summary

Ontology

  • Ontology changes in ontology/ are consistent with spec changes
  • New or modified terms are defined and documented

Converters

  • Converter logic in converters/ is updated to reflect spec or ontology changes
  • New converters include tests under the converter's test directory

Validation

  • Validation rules in validation/ are updated if the spec changed
  • New validation cases are covered by tests

Documentation

  • docs/ is updated to reflect any user-facing changes
  • New features or behaviors are documented with examples where appropriate
  • CONTRIBUTING.md is updated if the contribution process changed

Examples

  • examples/ are added or updated for any new spec constructs or converter support

Tests

  • All existing tests pass (pytest / CI green)
  • New functionality is covered by tests

Compliance

  • ASF license headers are present on all new source files
  • No third-party dependencies are added without PMC/IPMC approval

@MonkeyCanCode

Copy link
Copy Markdown
Contributor Author

@QMalcolm mind take a look?

Copilot AI lite review requested due to automatic review settings September 6, 2026 20:31

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The fix is small, localized, and includes a targeted regression test that confirms the discrete percentile flag is preserved through the round-trip.

Pull request overview

This PR fixes a DBT converter round-trip bug where PERCENTILE_DISC expressions were parsed and rebuilt as PERCENTILE_CONT, losing the discrete percentile flag during OSSIE ↔ MSI conversions.

Changes:

  • Extend _extract_agg_info() to preserve whether the parsed percentile aggregate is discrete (PERCENTILE_DISC) vs continuous (PERCENTILE_CONT).
  • Populate use_discrete_percentile in PydanticMeasureAggregationParameters when converting OSSIE → MSI.
  • Add a round-trip test ensuring use_discrete_percentile=True survives MSI → OSSIE → MSI.
File summaries
File Description
converters/dbt/src/ossie_dbt/expression_utils.py Preserve discrete-vs-continuous percentile info during aggregation parsing.
converters/dbt/src/ossie_dbt/ossie_to_msi.py Thread use_discrete_percentile into MSI aggregation params for percentile metrics.
converters/dbt/tests/test_ossie_to_msi.py Add regression test validating PERCENTILE_DISC round-trips correctly.
Review details
  • Files reviewed: 3/3 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +42 to 45
Returns ``(agg_type, bare_col, percentile, use_discrete_percentile)`` for recognised patterns,
``None`` otherwise. ``percentile`` is only set for ``PERCENTILE`` aggregations; it is ``None``
for all others. ``use_discrete_percentile`` is ``True`` only for ``PERCENTILE_DISC``.
The returned column name has any dataset qualifier stripped.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not in scope of this change. I can create a follow up to fix those doc-string later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants