Skip to content

Report a delegated test run as finished exactly once - #1900

Open
wenyt (wenytang-ms) wants to merge 1 commit into
developfrom
wenyt/delegate-test-reliability
Open

Report a delegated test run as finished exactly once#1900
wenyt (wenytang-ms) wants to merge 1 commit into
developfrom
wenyt/delegate-test-reliability

Conversation

@wenytang-ms

Copy link
Copy Markdown
Contributor

Why

A delegated test run could never end.

GradleDelegateCommandHandler dispatched buildTarget/test and returned immediately, so completion could only reach the client as a TEST_REPORT notification. The build server sends that report only once it has actually started a test run, and several paths end the request before that ever happens:

  • a Gradle version it cannot drive (< 2.6, or < 3.5 with environment variables) — it logs an error and returns without constructing a TestReportReporter
  • a GradleConnectionException / IllegalStateException from the outer connection
  • a target with nothing to run

In all of these the client is left waiting for a report that never comes: the spinner never stops and the test run stays pending until the window is reloaded.

Two more problems sit next to it:

  • Overlapping runs cross-report. Results arrive through extension-wide commands (java.gradle.buildServer.onDidChangeTestItemStatus / onDidFinishTestRun) that carry no run identity, and are resolved against the single bspContext the runner tracks. A second run started while the first is in flight reports into the first one's test items.
  • The init script is deleted while Gradle still needs it. For debug runs, launchWithBsp writes a per-run init script and removes it in a finally around the delegate command. Because the command returned on dispatch, the removal raced Gradle actually reading the script.

What

Wait for the result. GradleDelegateCommandHandler now joins buildTargetTest and returns the BSP status code. The end of the request becomes the authoritative end of the run, so a missing report degrades to a finished run instead of one that never ends. This also keeps the init script alive for the whole run — the debugger is now attached while the request is in flight, since the test JVM waits on the debug port for the duration of the run.

Report exactly once. reportFinished is idempotent, so the build server's own report still wins whenever it arrives, and the request completing is only a backstop.

Queue runs. launch chains onto the previous run and resolves on completion rather than dispatch. The chain survives a failed run, so one failure cannot block everything after it. A run cancelled while queued is reported without being launched, so the queue still advances.

Correlate reports with an origin id. The client attaches one to the test request; the build server already echoes it back on TestReport, so GradleBuildClient forwards it with the notification. A report belonging to a run that has already ended can no longer close the run that replaced it. Reports without an origin id are still accepted, so nothing regresses against an older build server.

Notes

  • IRunTestContext.cancellationToken is added to the mirrored java-test-runner.api.ts. It is optional: hosts released before it exists do not set it.
  • Status codes on the new paths use BSP's StatusCode values, matching what the notification path already sends. The XML fallback keeps its existing codes.

Testing

  • extension unit suite: 117 passing, including 6 new cases covering a missing report, report precedence, a stale report arriving during the next run, serialization, cancel-while-queued, and queue recovery after a failure.
  • jdtls.ext builds clean (mvn clean package).
  • tsc, eslint and prettier clean.

The delegate command dispatched the test request and returned, leaving
completion to arrive later as a build server notification. The build server
only reports a test run as finished once it has actually started one, so a
Gradle version it cannot drive, a connection that dropped and a target with
nothing to run all ended the request with no report at all, and the caller
waited for one forever.

Wait for the result instead, and treat the end of the request as the end of
the run. Reporting is idempotent, so the build server's own report still
wins whenever it arrives, and a lost report now degrades to a finished run
rather than a run that never ends. Waiting also keeps a per-run init script
alive until Gradle has read it, which the old dispatch-and-delete raced with;
the debugger is now attached while the request is in flight, since the test
JVM is waiting on the debug port for the duration of the run.

Runs are queued rather than interleaved. Results arrive through
extension-wide commands that carry no run identity and are resolved against
the single run this class tracks, so two overlapping runs reported into each
other's test items. The client now attaches an origin id that the build
server echoes back on the test report, so a report belonging to a run that
has already ended can no longer close the run that replaced it.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8b7a64f2-3b9f-4328-a173-f4728a1fa95c
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@wenytang-ms

Copy link
Copy Markdown
Contributor Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant