Add MockWebServer-based tests for BigqueryClient/BigqueryJavaOutputPlugin - #110
Open
chikamura wants to merge 8 commits into
Open
Add MockWebServer-based tests for BigqueryClient/BigqueryJavaOutputPlugin#110chikamura wants to merge 8 commits into
chikamura wants to merge 8 commits into
Conversation
Adds a test-only test_host config option that redirects BigqueryClient at a local MockWebServer instead of the real BigQuery API and bypasses credential setup. BigqueryClient#load() also routes load jobs through jobs.insert in this mode, since TableDataWriteChannel's resumable upload session hardcodes the real BigQuery host regardless of BigQueryOptions#getHost() (a google-cloud-bigquery:2.14.0 limitation). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds bounded RetrySettings for the test-host BigQuery client so retry tests don't wait out the library's real default backoff, refactors getBigQueryService() to apply setProjectId() once regardless of branch, and requires the TEST_HOST_ENABLED environment variable (set only by the `test` Gradle task) in addition to the test_host config, so a plugin config alone can no longer redirect a real embulk run at an arbitrary unauthenticated host. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…aOutputPlugin Introduces a shared MockWebServer test harness (BigqueryMockWebServerTestUtil, AssertUtil) and builds out coverage for BigqueryClient (retry behavior for load/copy/query/createTableIfNotExist, updateTableIfNeed's interaction with mode/retain flags/column_options) and BigqueryJavaOutputPlugin (an end-to-end run per mode against a mocked BigQuery API). Also consolidates duplicated request/response fixtures into the shared util as they emerged. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
merge_keys was only ever needed to keep testRunMergeModeWithMockWebServer from triggering the INFORMATION_SCHEMA merge-key lookup; set it there explicitly instead of baking it into every test's default config. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
is_skip_job_result_check defaulted to true in test_host.yml, so no MockWebServer test ever exercised getTransactionReport()'s row-count check or the temp table row-count lookup it performs. Flip it to false (matching the plugin's own production default) and update the affected tests: add outputRows to load job responses, enqueue the extra temp table GET, and shift request indices accordingly. Also clear BigqueryUtil's static FileWriterHolder map between tests (test-only: a real embulk run never calls transaction() twice in one JVM, but JUnit reuses the JVM/threads across test methods, which was inflating num_input_rows with stale writers from earlier tests). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Moves the MockWebServer-based tests out of TestBigqueryClient and TestBigqueryJavaOutputPlugin into new TestBigqueryClientWithMockServer and TestBigqueryJavaOutputPluginWithMockServer classes, leaving the original classes containing only their pre-existing, non-MockServer tests. Drops the now-redundant "WithMockWebServer" suffix from individual test method names, since the class name already says so, and updates a README reference to the class that used to own this behavior. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
chikamura
changed the title
chikamura
changed the title
chikamura
marked this pull request as ready for review
September 4, 2026 05:45
… host Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a MockWebServer-based test suite (
TestBigqueryClientWithMockServer,TestBigqueryJavaOutputPluginWithMockServer) forBigqueryClientandBigqueryJavaOutputPlugin. To let./gradlew testverify behavior without talking to the real BigQuery API, adds a way forBigqueryClientto connect to a test server, along with a new utility class for it,BigqueryTestHostSupport.java.test_hostconfig option. It only takes effect when theTEST_HOST_ENABLEDenvironment variable is also set, which in practice only happens under./gradlew test.test_hostis used for all ofBigqueryClient's BigQuery API calls.test_hostalso overridesload()'s behavior:google-cloud-bigquery:2.14.0ignorestest_host(BigQueryOptions#setHost()) for one ofload()'s connections and hardcodes it tohttps://www.googleapis.com, so this override works around that.TestBigqueryClientWithMockServerandTestBigqueryJavaOutputPluginWithMockServer, covering:createTableIfNotExistjobsupdateTableIfNeed()'s interaction betweenmode,retain_column_descriptions/retain_column_policy_tags, andcolumn_options[].descriptionappend,append_direct,replace,delete_in_advance,merge) against a mocked BigQuery APIcom.squareup.okhttp3:mockwebserver:4.11.0dependency andenvironment "TEST_HOST_ENABLED", "true"to thetesttask inbuild.gradle.Not in scope
Several known Ruby-vs-Java behavior differences are documented as
TODOcomments in the new tests (not fixed here):column_options[].descriptiononly applies tomode: replacein Java, whereas Ruby applies it regardless of moderetriesconfig is only consulted by the job-polling loops (load/copy/query); it has no effect on other API calls such ascreateTableIfNotExist(),getTable(),deleteTable(),createDataset(), orupdateTableIfNeed()'s PATCH — those rely on the underlying BigQuery client library's own default retry behavior insteadcreateTableIfNotExist()(Ruby strips it)This PR is test coverage only; those are left as follow-ups.
Test plan
./gradlew test— 104 tests pass./gradlew spotlessApply