Skip to content

[WIP] Guarantee temp table cleanup and fix its ordering vs schema update - #111

Closed
chikamura wants to merge 2 commits into
feature/mockwebserver-e2e-testsfrom
fix/temp-table-cleanup-order
Closed

[WIP] Guarantee temp table cleanup and fix its ordering vs schema update#111
chikamura wants to merge 2 commits into
feature/mockwebserver-e2e-testsfrom
fix/temp-table-cleanup-order

Conversation

@chikamura

@chikamura chikamura commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Ruby's begin/ensure wraps everything from right after auto_create through the final schema patch, and always deletes the temp table (and cleans up local files) afterward, so cleanup happens even if the load, copy/merge, or schema update fails. The java port only wrapped the copy/merge step, deleted the temp table inline before the schema update, and skipped it if a delete came before an unrelated failure.
  • Wrap the whole post-autoCreate body in try/finally so a failure anywhere (storeCachedSrcFieldsIfNeed, load, copy/merge, updateTableIfNeed) still guarantees the temp table delete, nest a second finally for the local intermediate file cleanup so a delete failure doesn't block it either, and move updateTableIfNeed() before the temp table delete so a delete failure no longer blocks the schema update from being attempted.
  • Updated the existing MockWebServer-based tests to match the new call order (schema update now happens before the temp table delete), and added testRunReplaceModeStillDeletesTempTableWhenStoreCachedSrcFieldsIfNeedFails to cover the temp table delete being guaranteed even when storeCachedSrcFieldsIfNeed() itself fails.

Test plan

  • ./gradlew test (105 tests, 1 skipped, all passing)
  • ./gradlew spotlessApply

Note

Much of the diff is just re-indentation from the new try/finally nesting — reviewing with "Hide whitespace changes" enabled is recommended.

🤖 Generated with Claude Code

Ruby's begin/ensure wraps everything from right after auto_create
through the final schema patch, and always deletes the temp table (and
cleans up local files) afterward, so cleanup happens even if the load,
copy/merge, or schema update fails. The java port only wrapped the
copy/merge step, deleted the temp table inline before the schema
update, and skipped it if a delete came before an unrelated failure.

Wrap the whole post-autoCreate body in try/finally so a failure
anywhere (storeCachedSrcFieldsIfNeed, load, copy/merge,
updateTableIfNeed) still guarantees the temp table delete, nest a
second finally for the local intermediate file cleanup so a delete
failure doesn't block it either, and move updateTableIfNeed() before
the temp table delete so a delete failure no longer blocks the schema
update from being attempted.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
client.updateTableIfNeed();
} finally {
try {
if (task.getTempTable().isPresent()) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the mode switch since it was redundant: BigqueryTaskBuilder.setTempTable() only auto-populates temp_table for exactly these four modes, so task.getTempTable().isPresent() alone already implies the same restriction. This also matches ruby's own unconditional if task['temp_table'] check in its ensure block, with no mode restriction.

https://github.com/trocco-io/embulk-output-bigquery/blob/b4f8d2f4baf1c431a1cee3631790526441c3f9ec/lib/embulk/output/bigquery.rb#L498-L500

transaction() returns right after creating the destination table when
paths.isEmpty(), before ever reaching the load/copy/updateTableIfNeed
logic. Add a regression test covering that the temp table (already
created by autoCreate()) still gets deleted via the guaranteed finally
block in that case, using zero input records to reproduce
paths.isEmpty() (BigqueryPageOutput#add() is never called, so no
writer is registered and no intermediate file is ever created).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@chikamura chikamura closed this Sep 9, 2026
@chikamura
chikamura deleted the fix/temp-table-cleanup-order branch September 9, 2026 04:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant