Skip to content

Fix: generate_surrogate_key returns hex strings for SHA256/SHA512 on Presto and Trino#5888

Open
Pawansingh3889 wants to merge 1 commit into
SQLMesh:mainfrom
Pawansingh3889:fix/surrogate-key-sha2-hex
Open

Fix: generate_surrogate_key returns hex strings for SHA256/SHA512 on Presto and Trino#5888
Pawansingh3889 wants to merge 1 commit into
SQLMesh:mainfrom
Pawansingh3889:fix/surrogate-key-sha2-hex

Conversation

@Pawansingh3889

@Pawansingh3889 Pawansingh3889 commented Jul 7, 2026

Copy link
Copy Markdown

Description

Fixes #5871.

@GENERATE_SURROGATE_KEY(..., hash_function := 'SHA256') produces a key that changes value depending on the engine. With MD5 the macro converts the parsed exp.MD5Digest into exp.MD5, so Presto and Trino render the hex-string form (LOWER(TO_HEX(MD5(TO_UTF8(...))))). With SHA256/SHA512 there is no equivalent conversion, so Trino gets a bare SHA256(varchar): a type error at execution time, and a binary digest rather than a hex string where it does run.

The upstream half merged in sqlglot this week (tobymao/sqlglot#7824): Presto/Trino now map native SHA256/SHA512 to exp.SHA2Digest and render exp.SHA2 in the hex-string form, mirroring their MD5/MD5Digest handling.

This PR is the macro half, mirroring the MD5 conversion that already sits one line above:

  • exp.SHA2Digest results convert to exp.SHA2 (keeping the length), so the surrogate key is always the string-semantics expression.
  • The concatenated argument is annotated as text, which lets generators that wrap an encode around string inputs (TO_UTF8 on Presto/Trino) do so without a separate annotation pass.

Behavior on the currently pinned sqlglot (30.8.0) is byte-identical for every dialect; I verified the rendered SQL for MD5/SHA256 across trino, duckdb, bigquery and snowflake before and after. The conversion itself is still exercised today because BigQuery's parser already maps SHA256 to SHA2Digest. Once the sqlglot pin moves past #7824, Trino and Presto start rendering LOWER(TO_HEX(SHA256(TO_UTF8(...)))) with no further change here; the string assertions for that are noted in the test for whoever does the bump (happy to follow up with them at that point).

Test Plan

  • New test_generate_surrogate_key_hash_semantics in tests/core/test_macros.py: asserts the macro returns exp.SHA2 (never a digest node) with a text-typed argument, and pins the rendered SQL for bigquery/duckdb SHA256 and trino MD5, all stable across supported sqlglot versions.
  • Full tests/core/test_macros.py: 139 passed.
  • Manual render matrix (MD5/SHA256/SHA512 across trino, presto, duckdb, bigquery, snowflake, spark) on sqlglot 30.8.0 before and after the change: byte-identical output. Same matrix against sqlglot main: trino/presto render the hex-string form, other dialects unchanged.

Checklist

  • I have run make style and fixed any issues (ruff check and ruff format pass on the changed files; happy to fix anything else CI flags)
  • I have added tests for my changes (if applicable)
  • All existing tests pass (make fast-test) - ran tests/core/test_macros.py in full (139 passed); leaving the broader suite to CI
  • My commits are signed off (git commit -s) per the DCO

…Presto and Trino

Signed-off-by: Pawan Singh Kapkoti <42340841+Pawansingh3889@users.noreply.github.com>
@Pawansingh3889 Pawansingh3889 force-pushed the fix/surrogate-key-sha2-hex branch from a6384fe to f580eb3 Compare July 7, 2026 14:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG [TRINO]: @generate_surrogate_key macro does not work with SHA256 on trino

1 participant