Skip to content

fix: resolve @property provenance through CTE bodies, correct-or-silent - #243

Merged
Marius Volkhart (MariusVolkhart) merged 1 commit into
mainfrom
fix/229-cte-expression-property-kdoc
Aug 22, 2026
Merged

fix: resolve @property provenance through CTE bodies, correct-or-silent#243
Marius Volkhart (MariusVolkhart) merged 1 commit into
mainfrom
fix/229-cte-expression-property-kdoc

Conversation

@MariusVolkhart

@MariusVolkhart Marius Volkhart (MariusVolkhart) commented Aug 22, 2026

Copy link
Copy Markdown
Member

Fixes #229.

A nullable result-class property shipped with no @property KDoc line at all when it was both CTE-wrapped and expression-derived. parseSelectItems deliberately windows past the WITH clause, so the outer item is a plain reference, and pgjdbc reports an empty table name for such a column — both inputs to sourceReference() were empty, so the line was filtered out entirely.

-- before: no @property line for description_upper
WITH deleted_parent AS (
  DELETE FROM parent WHERE id = ? RETURNING id, name, UPPER(description) AS description_upper
)
SELECT id, name, description_upper FROM deleted_parent;

Provenance now resolves through the CTE definition rather than the outer select list.

The design is a whitelist, and that is the point

A general resolver was written first — multi-CTE, alias-addressed, JOIN-aware, DML main queries. It emitted wrong provenance across a long tail of valid SQL. Five adversarial verification rounds, each probing the real function and checking answers against a live PostgreSQL 18, each closed the reported defects and found new ones:

  • a CTE with an explicit column list was invisible to the sibling-ambiguity scan, so another CTE's expression was attributed to a column it never produced
  • a FROM alias shadowing a different CTE's name resolved to the wrong CTE
  • a DML target relation was treated as a CTE source (Postgres resolves DELETE FROM x to the real table, never a CTE)
  • INSERT ... SELECT ... FROM cte RETURNING resolved against the feeding FROM instead of the insert target
  • top-level set operations reported only the first branch's expression
  • missing identifier folding, comment text leaking into the emitted expression, bare reserved keywords matched as columns

A wrong @property x (`SOME_EXPR`) is worse than a missing line, so resolution is now restricted to a shape whose correctness argument fits in one sentence: exactly one CTE declared, and the main query's FROM is exactly that CTE. Under that precondition sibling ambiguity cannot arise at all, and ~400 lines of ambiguity/address-map/JOIN machinery was deleted rather than gated. Everything else returns null and reproduces the old behaviour — a missing line, never a wrong one.

PgCatalogLoader already states the principle this converged on: string-based SQL transformation cannot be proven exhaustively correct for every input shape.

Shapes outside the whitelist are tracked in #238, which also records why a bigger hand-written parser is the wrong direction and what was already investigated on the Postgres-authoritative path.

Two pre-existing wrong outputs fixed on the way

Fixing the reported trailing-comment and quoted-alias gaps required widening COLUMN_REFERENCE to match quoted identifiers. That corrects two existing defects, both proven by building the generator at the merge base and running the same inputs through it:

Query Before After
SELECT "Foo" AS bar FROM tq (`tq.bar`), comment silently dropped (`tq.Foo`) with its Postgres column comment
quoted CTE alias (`"descriptionUpper"`) — echoed its own name (`UPPER(description)`)

The first one mattered: originalName came from pgjdbc's getColumnName, which returns the alias, so catalog.findColumn looked up a column that does not exist and quietly lost the comment.

Generating the space-containing-name test case also surfaced a latent bug — @property My Col emitted two bare tokens, corrupting the tag. formatAsKdocPropertyReference now backticks names that are not plain identifiers. Backticks are the correct escape here, confirmed by reading KDocName.getNameTextRange rather than assuming; [My Col] would not bind.

Postgres parity details worth a reviewer's eye

  • Identifier folding is ASCII-only, matching downcase_identifier. Kotlin's String.lowercase() applies full Unicode mapping and disagrees with Postgres: SELECT Ü matches a column named "ü" under Unicode folding, but Postgres raises column "Ü" does not exist. There is a comment on foldAsciiCase saying not to "fix" it back.
  • PG_RESERVED_KEYWORDS is pg_get_keywords() categories R + T, verified 101/101 against live PG 16/17/18. Deliberately wider than the keywords currently known to be niladic: a reserved word that is not niladic cannot appear as a bare select-list item at all, so including it costs nothing and absorbs future additions (as system_user was in PG 16, which is exactly how this bug class reappeared once).
  • Quoted-identifier scanning is ""-escape aware and shares one pattern with parseColumnReference, since a divergence between two scanners is what produced an earlier cross-match.

Verification

  • 1377 tests green, including 133 unchanged ParseSelectItems cases
  • parseSelectItems compared at the merge base vs this branch across all 185 scenario statements: 181 identical, 4 differing, all intended quoted references
  • the resolver probed across ~130 SQL shapes with every non-null answer checked against live PostgreSQL 18.4; non-ASCII folding covered for Ü/ü, Turkish İ/ı, ß, Greek final sigma, full-width , Cyrillic А, and combining vs precomposed forms
  • key fixes mutation-checked: swapping foldAsciiCase for lowercase() flips two answers to Postgres-contradicting values
  • golden regeneration is stable and produced no drift after rebasing onto the upstream GROUPING SETS nullability change
  • generated goldens compile

./gradlew build passes. (An earlier note here claimed a pre-existing e2e-tests-micronaut failure; that was my local Gradle daemon running JDK 21 against a build that declares JDK 25, not a repo problem. Filed as #246, since the daemon JVM is unenforced and the resulting error is cryptic.)

🤖 Generated with Claude Code

A nullable result-class property shipped with no @Property KDoc line at all
when it was both CTE-wrapped and expression-derived. `parseSelectItems`
deliberately windows past the WITH clause, so the outer item is a plain
reference; pgjdbc reports an empty table name for such a column. Both inputs
to `sourceReference()` were empty, so the line was filtered out entirely --
exactly the case the project's "document what null means" rule exists for.

Provenance now resolves through the CTE definition rather than the outer
select list, via `resolveCteOutputExpression`.

The design is a whitelist, not a blacklist, and that choice is load-bearing.
A general resolver was written first -- multi-CTE, alias-addressed,
JOIN-aware, DML main queries -- and it emitted *wrong* provenance across a
long tail of valid SQL: column-list CTEs invisible to the ambiguity scan,
FROM aliases shadowing another CTE's name, DML target relations treated as
CTE sources, INSERT RETURNING resolving against the feeding FROM instead of
the insert target, set operations reporting only the first branch. A wrong
`@property x (`SOME_EXPR`)` is worse than a missing line, so resolution is
now restricted to a shape whose correctness argument is one sentence: exactly
one CTE declared, and the main query's FROM is exactly that CTE. Under that
precondition sibling ambiguity cannot arise. Everything else returns null and
reproduces the old behaviour -- a missing line, never a wrong one.
PgCatalogLoader states the same principle: string-based SQL transformation
cannot be proven exhaustively correct for every input shape.

Fixing the reported trailing-comment and quoted-alias gaps required widening
`COLUMN_REFERENCE` to match quoted identifiers, which corrects two
pre-existing wrong outputs on the shared path. `SELECT "Foo" AS bar` derived
`originalName` from pgjdbc's `getColumnName` -- the alias `bar` -- so
`catalog.findColumn` looked up a column that does not exist and silently
dropped the Postgres column comment; it now resolves through the quoted
source. A quoted CTE alias echoed its own name back as its provenance
instead of reporting the expression.

Identifier folding is ASCII-only, matching `downcase_identifier`. Kotlin's
`String.lowercase()` applies full Unicode mapping, which disagrees with
Postgres about which column an unquoted non-ASCII reference targets --
`SELECT Ü` matches a column named "ü" under Unicode folding but not in
Postgres.

Mechanical changes:

- `resolveCteOutputExpression` in SqlUtils.kt, called from TypeRepository only
  where `column.table == null && selectItem.columnName != null`
- `parseAliasToken`: reads one identifier token after AS, escape-aware for
  quoted names, rejecting trailing content rather than discarding it
- `COLUMN_REFERENCE` matches quoted identifiers; `SelectItem` carries
  `isColumnNameQuoted`/`isTableNameQuoted` so folding can distinguish
  quoted-exact from unquoted-ASCII-folded comparisons
- `foldAsciiCase`, plus `PG_RESERVED_KEYWORDS` (pg_get_keywords R+T) so a bare
  reserved word is never matched as a column
- `formatAsKdocPropertyReference`: backtick-wraps @Property names that are not
  plain identifiers. A space-containing name previously emitted two bare
  tokens, corrupting the tag
- golden coverage for quoted column references, including a quoted column
  carrying a Postgres comment

Scenarios beyond the single-CTE whitelist are tracked in #238.

Fixes #229

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@snyk-io

snyk-io Bot commented Aug 22, 2026

Copy link
Copy Markdown

Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
Open Source Security 0 0 0 0 0 issues
Licenses 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

@MariusVolkhart
Marius Volkhart (MariusVolkhart) merged commit b41b813 into main Aug 22, 2026
13 checks passed
@MariusVolkhart
Marius Volkhart (MariusVolkhart) deleted the fix/229-cte-expression-property-kdoc branch August 22, 2026 15:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

@property KDoc line dropped for columns that are both CTE-wrapped and expression-derived

1 participant