Skip to content

Clean up Windows action process trees - #246

Open
alex-the-third wants to merge 1 commit into
buildbarn:mainfrom
alex-the-third:fix/windows-process-cleanup
Open

Clean up Windows action process trees#246
alex-the-third wants to merge 1 commit into
buildbarn:mainfrom
alex-the-third:fix/windows-process-cleanup

Conversation

@alex-the-third

Copy link
Copy Markdown

On Windows, actions may exit while leaving subprocesses running. Those subprocesses can retain their working directory or open files inside the action root, causing the executor to fail when removing it after the action completes.

This implements the approach proposed in #245 and follows Bazel's native Windows launcher.

Implementation

Each action process is placed in its own non-breakaway Windows Job Object configured with JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE.

The root process is created suspended, assigned to the job, and only then resumed. This prevents it from spawning descendants before job assignment. Because Go's os/exec closes the primary thread handle before cmd.Start() returns, the implementation locates the root process's threads by PID and resumes them through the Windows API.

When the root process exits, any remaining processes in the job are terminated. The runner then waits for JOB_OBJECT_MSG_ACTIVE_PROCESS_ZERO before returning, ensuring that descendants no longer hold files or directories inside the action root.

Command cancellation terminates the entire job instead of only the root process. Startup cancellation is serialized with process resumption so cancellation cannot terminate the suspended process between the cancellation check and resume.

Non-Windows behavior remains unchanged.

Testing

Added Windows tests covering:

  • A parent that exits while leaving a child holding a file open inside the action root.
  • Cancellation of a running parent and descendant.
  • The race between the startup cancellation check and resuming the suspended process.
  • Immediate removal of the action root after Run() returns.

Fixes #245

Create action processes suspended and assign them to non-breakaway job objects before allowing them to execute. Terminate and wait for surviving descendants when the root process exits so build directory cleanup does not race lingering processes.

Serialize startup cancellation with process resumption and cover descendant cleanup and the cancellation race with Windows tests.
@EdSchouten

Copy link
Copy Markdown
Member

Though I appreciate the work that went into this change, I think that this is not something we should try to tackle on the Buildbarn side. In my opinion it is a feature that should be supported natively by Go's os/exec API. Just like how it supports chroot() on UNIX. In fact, there is currently work under way to get this added:

golang/go#79927
golang/go#80415

Can you please work with the authors of those issues to validate that all of the features you need are part of those changes? Thanks!

response, err := runner.Run(context.Background(), &runner_pb.RunRequest{
Arguments: []string{
testBinaryPath,
"-test.run=TestLocalRunnerRunWindowsSubprocessCleanupHelper",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's safe to assume that you can do this. What happens if rules_go's go_test() is restructured to compile Go test binaries with a different type of entry point?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, agree that we should probably keep the implementation details of rules_go out of here, though I doubt that it is likely to change...
Anyhow, I could either move the helper dispatch into TestMain as in (rules_go #4335/#4336) or use a dedicated go_binary.

@alex-the-third

Copy link
Copy Markdown
Author

I tested Buildbarn against the actual implementation in Go CL 801640. It satisfies the launch-time containment requirements and lets me remove CREATE_SUSPENDED, post-start AssignProcessToJobObject, thread enumeration/resumption, and the associated startup cancellation synchronization.
The Windows implementation still works as desired but as you suspected can be done a lot leaner with it.

The remaining code is the Buildbarn-specific action lifecycle: creating/configuring the per-action Job Object, whole-tree cancellation, terminating descendants after the root exits, waiting for ACTIVE_PROCESS_ZERO, and cleanup.
I think this is acceptable because to me those are executor semantics and not generics.

I found one Go API edge case (Jobs: []Handle{} gives ERROR_BAD_LENGTH, unlike nil) and still need clarification.

You can close/defer this PR until Go is ready - we are not pressured by time here and I just continue using my patched executor.

Also if you want I can publish a prototype of the stuff that would remain after the Go API change.

@EdSchouten

Copy link
Copy Markdown
Member

Awesome! Yeah, if you want, you can just push the eventual changes in this PR and switch it back to draft. Then by the time Go 1.28 is out, all you need to do is move it out of draft again.

Thanks for looking into this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Windows: remote executor fails to delete build root when actions leave subprocesses running

2 participants