Clean up Windows action process trees - #246
Conversation
Create action processes suspended and assign them to non-breakaway job objects before allowing them to execute. Terminate and wait for surviving descendants when the root process exits so build directory cleanup does not race lingering processes. Serialize startup cancellation with process resumption and cover descendant cleanup and the cancellation race with Windows tests.
|
Though I appreciate the work that went into this change, I think that this is not something we should try to tackle on the Buildbarn side. In my opinion it is a feature that should be supported natively by Go's golang/go#79927 Can you please work with the authors of those issues to validate that all of the features you need are part of those changes? Thanks! |
| response, err := runner.Run(context.Background(), &runner_pb.RunRequest{ | ||
| Arguments: []string{ | ||
| testBinaryPath, | ||
| "-test.run=TestLocalRunnerRunWindowsSubprocessCleanupHelper", |
There was a problem hiding this comment.
I don't think it's safe to assume that you can do this. What happens if rules_go's go_test() is restructured to compile Go test binaries with a different type of entry point?
There was a problem hiding this comment.
Hmm, agree that we should probably keep the implementation details of rules_go out of here, though I doubt that it is likely to change...
Anyhow, I could either move the helper dispatch into TestMain as in (rules_go #4335/#4336) or use a dedicated go_binary.
|
I tested Buildbarn against the actual implementation in Go CL 801640. It satisfies the launch-time containment requirements and lets me remove The remaining code is the Buildbarn-specific action lifecycle: creating/configuring the per-action Job Object, whole-tree cancellation, terminating descendants after the root exits, waiting for I found one Go API edge case ( You can close/defer this PR until Go is ready - we are not pressured by time here and I just continue using my patched executor. Also if you want I can publish a prototype of the stuff that would remain after the Go API change. |
|
Awesome! Yeah, if you want, you can just push the eventual changes in this PR and switch it back to draft. Then by the time Go 1.28 is out, all you need to do is move it out of draft again. Thanks for looking into this! |
On Windows, actions may exit while leaving subprocesses running. Those subprocesses can retain their working directory or open files inside the action root, causing the executor to fail when removing it after the action completes.
This implements the approach proposed in #245 and follows Bazel's native Windows launcher.
Implementation
Each action process is placed in its own non-breakaway Windows Job Object configured with
JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE.The root process is created suspended, assigned to the job, and only then resumed. This prevents it from spawning descendants before job assignment. Because Go's
os/execcloses the primary thread handle beforecmd.Start()returns, the implementation locates the root process's threads by PID and resumes them through the Windows API.When the root process exits, any remaining processes in the job are terminated. The runner then waits for
JOB_OBJECT_MSG_ACTIVE_PROCESS_ZERObefore returning, ensuring that descendants no longer hold files or directories inside the action root.Command cancellation terminates the entire job instead of only the root process. Startup cancellation is serialized with process resumption so cancellation cannot terminate the suspended process between the cancellation check and resume.
Non-Windows behavior remains unchanged.
Testing
Added Windows tests covering:
Run()returns.Fixes #245