Skip to content

[HSTACK] feat: added codec for DeltaScanMetaExec - #22

Closed
mihai-bleont wants to merge 11 commits into
main-1.0-statsfrom
meta_codec
Closed

[HSTACK] feat: added codec for DeltaScanMetaExec#22
mihai-bleont wants to merge 11 commits into
main-1.0-statsfrom
meta_codec

Conversation

@mihai-bleont

@mihai-bleont mihai-bleont commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

This is mostly copied from the DeltaScanExec codec. The important difference is that DeltaScanMetaExec has no child plan, instead it has a pair vector of (file_id, row_count) embedded.

I tested with the following SQLs that used to fail:

SELECT _ACP_DATE, COUNT(*) FROM aep_dataset_next('4E9432245BC7C44B0A494037@AdobeOrg', 'prod', '', '68e0baa39036635052fa02cd') GROUP BY _ACP_DATE;
SELECT DISTINCT _ACP_DATE FROM aep_dataset_next('4E9432245BC7C44B0A494037@AdobeOrg', 'prod', '', '68e0baa39036635052fa02cd');
SELECT _ACP_DATE FROM aep_dataset_next('4E9432245BC7C44B0A494037@AdobeOrg', 'prod', '', '68e0baa39036635052fa02cd');
SELECT _ACP_DATE, COUNT(*) FROM aep_dataset_next('4E9432245BC7C44B0A494037@AdobeOrg', 'prod', '', '68e0baa39036635052fa02cd')
WHERE _ACP_DATE = '2025-10-04' GROUP BY _ACP_DATE;

SELECT r.timestamp, r._id, n._ACP_DATE, n.cnt
FROM (
  SELECT timestamp, _id
  FROM aep_dataset('4E9432245BC7C44B0A494037@AdobeOrg','prod','','abfss://gen1@datalakeg462p5ax3pjbzvdp.dfs.core.windows.net/platform/5fc8afc657486818dd2c03ed-v1/')
  LIMIT 1
) r
CROSS JOIN (
  SELECT _ACP_DATE, COUNT(*) AS cnt
  FROM aep_dataset_next('4E9432245BC7C44B0A494037@AdobeOrg', 'prod', '', '68e0baa39036635052fa02cd')
  GROUP BY _ACP_DATE
) n

@github-actions

Copy link
Copy Markdown

ACTION NEEDED

delta-rs follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

@aditanase aditanase left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mihai-bleont have we considered pushing this change upstream? (together with the other one? cc @adragomir )

// The only thing that we set in it is the log_size_limiter -
// which has already been used early in the logical / planning phase
// At upgrade, RECHECK usage sites for DeltaTableConfig, we'll need to re-evaluate if
// stuff begins writing to it

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we'll also use "with files" boolean flag pretty soon

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit out of context with one. @aditanase is there anything we want to address in this PR?

})?;

let wire = DeltaScanMetaExecWire::try_from(meta_scan)?;
serde_json::to_writer(buf, &wire).map_err(|e| {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we know how big this is? in general, do we have a metric for the plan size?
Ideally this would be extremely small, relying solely on metadata and with values burned in, I'd like to check though that for a really large table we are not carrying over all of the metadata in JSON form

@mihai-bleont

Copy link
Copy Markdown
Collaborator Author

This has been merged in c76cb33

@mihai-bleont
mihai-bleont deleted the meta_codec branch August 27, 2026 13:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants