[WIP] Guarantee temp table cleanup and fix its ordering vs schema update - #111
Closed
chikamura wants to merge 2 commits into
Closed
[WIP] Guarantee temp table cleanup and fix its ordering vs schema update#111chikamura wants to merge 2 commits into
chikamura wants to merge 2 commits into
Conversation
Ruby's begin/ensure wraps everything from right after auto_create through the final schema patch, and always deletes the temp table (and cleans up local files) afterward, so cleanup happens even if the load, copy/merge, or schema update fails. The java port only wrapped the copy/merge step, deleted the temp table inline before the schema update, and skipped it if a delete came before an unrelated failure. Wrap the whole post-autoCreate body in try/finally so a failure anywhere (storeCachedSrcFieldsIfNeed, load, copy/merge, updateTableIfNeed) still guarantees the temp table delete, nest a second finally for the local intermediate file cleanup so a delete failure doesn't block it either, and move updateTableIfNeed() before the temp table delete so a delete failure no longer blocks the schema update from being attempted. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
chikamura
commented
Sep 8, 2026
| client.updateTableIfNeed(); | ||
| } finally { | ||
| try { | ||
| if (task.getTempTable().isPresent()) { |
Contributor
Author
There was a problem hiding this comment.
Removed the mode switch since it was redundant: BigqueryTaskBuilder.setTempTable() only auto-populates temp_table for exactly these four modes, so task.getTempTable().isPresent() alone already implies the same restriction. This also matches ruby's own unconditional if task['temp_table'] check in its ensure block, with no mode restriction.
transaction() returns right after creating the destination table when paths.isEmpty(), before ever reaching the load/copy/updateTableIfNeed logic. Add a regression test covering that the temp table (already created by autoCreate()) still gets deleted via the guaranteed finally block in that case, using zero input records to reproduce paths.isEmpty() (BigqueryPageOutput#add() is never called, so no writer is registered and no intermediate file is ever created). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
begin/ensurewraps everything from right afterauto_createthrough the final schema patch, and always deletes the temp table (and cleans up local files) afterward, so cleanup happens even if the load, copy/merge, or schema update fails. The java port only wrapped the copy/merge step, deleted the temp table inline before the schema update, and skipped it if a delete came before an unrelated failure.autoCreatebody intry/finallyso a failure anywhere (storeCachedSrcFieldsIfNeed, load, copy/merge,updateTableIfNeed) still guarantees the temp table delete, nest a secondfinallyfor the local intermediate file cleanup so a delete failure doesn't block it either, and moveupdateTableIfNeed()before the temp table delete so a delete failure no longer blocks the schema update from being attempted.testRunReplaceModeStillDeletesTempTableWhenStoreCachedSrcFieldsIfNeedFailsto cover the temp table delete being guaranteed even whenstoreCachedSrcFieldsIfNeed()itself fails.Test plan
./gradlew test(105 tests, 1 skipped, all passing)./gradlew spotlessApplyNote
Much of the diff is just re-indentation from the new
try/finallynesting — reviewing with "Hide whitespace changes" enabled is recommended.🤖 Generated with Claude Code