[HSTACK] feat: added codec for DeltaScanMetaExec - #22
Conversation
…onPlan: DeltaNextPhysicalCodec
…e some fields nullable
… truncation was applied or not
|
ACTION NEEDED delta-rs follows the Conventional Commits specification for release automation. The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. |
aditanase
left a comment
There was a problem hiding this comment.
@mihai-bleont have we considered pushing this change upstream? (together with the other one? cc @adragomir )
| // The only thing that we set in it is the log_size_limiter - | ||
| // which has already been used early in the logical / planning phase | ||
| // At upgrade, RECHECK usage sites for DeltaTableConfig, we'll need to re-evaluate if | ||
| // stuff begins writing to it |
There was a problem hiding this comment.
I think we'll also use "with files" boolean flag pretty soon
There was a problem hiding this comment.
I'm a bit out of context with one. @aditanase is there anything we want to address in this PR?
| })?; | ||
|
|
||
| let wire = DeltaScanMetaExecWire::try_from(meta_scan)?; | ||
| serde_json::to_writer(buf, &wire).map_err(|e| { |
There was a problem hiding this comment.
do we know how big this is? in general, do we have a metric for the plan size?
Ideally this would be extremely small, relying solely on metadata and with values burned in, I'd like to check though that for a really large table we are not carrying over all of the metadata in JSON form
5c2d792 to
5a25b6e
Compare
|
This has been merged in c76cb33 |
This is mostly copied from the DeltaScanExec codec. The important difference is that DeltaScanMetaExec has no child plan, instead it has a pair vector of (file_id, row_count) embedded.
I tested with the following SQLs that used to fail: