Skip to content

fix(agent): honor RuntimeProcessOptions.stdin in the container runtime (PRI-3250) - #415

Merged
obra merged 3 commits into
obra:mainfrom
ada-sen:fix-pri-3250-container-runtime-stdin
Sep 26, 2026
Merged

obra merged 3 commits into
obra:mainfrom
ada-sen:fix-pri-3250-container-runtime-stdin

Conversation

@ada-sen

@ada-sen ada-sen commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Based on main. The stdin option this builds on landed in #418 (PRI-3243); the process-group kill work is deferred to #413 and is not part of this PR.

Problem

#418 gave RuntimeProcessOptions a stdin?: 'ignore' | 'pipe' option, defaulted to 'ignore', so a bash spawn doesn't hang on a command that unexpectedly reads stdin (e.g. head with no file args). That only reaches the host runtime (HostProcessRunner.start() in host.ts); types.ts said so explicitly.

The container runtime's ProjectedContainerProcessRunner.start() (packages/agent/src/tools/runtime/projected-container.ts) never read opts.stdin. It always calls containerManager.execStream(), which spawns its docker/container/plane CLI with -i and a piped stdin (see execStream() in docker-container.ts, apple-container.ts, plane-runtime.ts). Nothing wrote to or closed that pipe, so a container-run command that fell back to reading stdin would hang forever. Same shape as PRI-3243, one layer down. This is PRI-3250.

Fix

  • ProjectedContainerProcessRunner.start() ends the execStream handle's stdin immediately (with an error listener to swallow EPIPE) unless the caller passed stdin: 'pipe'. The returned RuntimeProcessHandle.stdin is undefined in that case, so a caller can't write to an ended stream.
  • Every container-mode caller that writes stdin now opts in:
    • container-exec-fs.ts writeTextFile passes stdin: 'pipe', and kills the process and throws if the pipe is still missing.
    • container-exec-network.ts fetch passes stdin: 'pipe' only when there is a body, and kills curl and throws if the pipe is missing, instead of silently sending an empty body.
  • Callers audited and unaffected: bash.ts and shell-job.ts pass stdin: 'ignore'; runtime-stdio-transport.ts (in-container MCP servers) already passes stdin: 'pipe'; stat/readTextFile/mkdir/readdir, file_find.ts and ripgrep_search.ts use exec() and never write stdin.
  • types.ts: the RuntimeProcessOptions.stdin doc now says every runtime honors it.

Tests

  • projected-container.test.ts
    • ends the container stdin pipe immediately when the caller does not opt into it: end() called once, stream writableEnded, returned handle.stdin undefined. Fails against the pre-fix runner.
    • does not end the container stdin pipe when the caller opts into stdin: pipe: guards against ending or hiding the pipe for 'pipe' callers, including writing through the returned handle. A runner that ignored opts.stdin would also pass this one; the test above is what requires 'ignore' to be honored.
    • container stdin writers composed with the real runner: the real ProjectedContainerProcessRunner wired to the real ContainerExecFileSystem / ContainerExecNetworkClient. writeTextFile content and a POST body both arrive over the container stdin pipe. Both fail if either call site drops stdin: 'pipe'.
  • container-exec-fs.test.ts / container-exec-network.test.ts: the fakes hand back a live stdin only for stdin: 'pipe', like the real runners. They can also withhold stdin and count kill() calls, and new tests cover the fail-loud branches: writeTextFile and a body-bearing fetch each kill the process and throw ... write stream unavailable, and a body-less fetch works without stdin. Reverting either guard to a silent skip fails its test.

This round

Gates on the pushed head: npm run build, npm run lint, npm run format:check, npm run typecheck, and npm run test --workspace=packages/agent (3833 passed, 27 skipped; 283 passed, 12 skipped in the serial e2e pass).

@obra

obra commented Sep 25, 2026 •

Copy link
Copy Markdown
Owner

Review from jc: competing adversarial reviewers, then a verifier that tried to refute every finding against the PR head and current origin/main. Verdict: changes-needed.

The underlying gap still exists on origin/main: projected-container.ts:428 still returns containerHandle.stdin unconditionally, and #413 is still open, so the fix is still needed. As committed, though, it breaks every container file_write/file_edit and silently empties url_fetch POST bodies, because the only two production stdin writers (container-exec-fs.ts:82 and container-exec-network.ts:49) never pass stdin:'pipe'. The #413 review flagged exactly this, and the PR body wrongly claims those paths are unaffected. The fix is small: add stdin:'pipe' at both call sites, add composed tests, and correct the body. It merges cleanly onto origin/main.

Confirmed findings:

  1. [critical, fix: small] packages/agent/src/tools/runtime/projected-container.ts:402-408,446 with container-exec-fs.ts:82-86
    All container-mode file writes break. ContainerExecFileSystem.writeTextFile calls this.process.start(['sh','-c','base64 -d > "$0"', p]) without stdin:'pipe'. The runner now ends the pipe and returns stdin: undefined, so writeTextFile throws 'ContainerExecFileSystem write stream unavailable'. That breaks file_write and file_edit for every projected-container runtime. The fix(agent): reap detached bash process groups within the parent's kill window (PRI-3243) #413 review warned about exactly this call site.
    Why it's real: I checked the PR head (21eb956). projected-container.ts:516 wires this.fs = new ContainerExecFileSystem(this.process), where this.process is the modified ProjectedContainerProcessRunner. container-exec-fs.ts:82 passes no opts, so opts.stdin defaults to 'ignore'. :446 then returns stdin undefined and :83-85 throws. Two reviewers independently reproduced it through the real runtime and saw it pass with the fix reverted.
    Suggested fix: Pass { stdin: 'pipe' } to the start() call in writeTextFile. Add a test that composes ProjectedContainerToolRuntime (fake execStream manager) with runtime.fs.writeTextFile, so the fs layer is exercised through the real runner.
  2. [critical, fix: small] packages/agent/src/tools/runtime/projected-container.ts:402-408,446 with container-exec-network.ts:49-56 (origin/main :119-125)
    Container-mode POST bodies are silently dropped. ContainerExecNetworkClient.fetch starts curl with --data-binary @- and no stdin:'pipe'. handle.stdin is now undefined, so the if (handle.stdin) guard skips the body write. curl reads EOF and sends an empty body, and no error is raised.
    Why it's real: projected-container.ts:517 wires this.network = new ContainerExecNetworkClient(this.process). container-exec-network.ts:49 calls start(argv, { signal }) with no stdin option. On origin/main the start call and guard are unchanged, and git merge-tree against origin/main is clean, so the regression survives a rebase. Both reviewers reproduced '' being written in place of the body.
    Suggested fix: Pass { signal: opts?.signal, stdin: 'pipe' } in fetch. Consider replacing the silent if (handle.stdin) skip with a throw when hasBody is set and stdin is missing, as writeTextFile does. Add a composed runtime.network.fetch POST test.
  3. [important, fix: trivial] PR #415 body paragraph 3 / commit 21eb956cb message
    The PR body falsely says ContainerExecFileSystem/ContainerExecNetworkClient have their 'own low-level exec-with-stdin paths ... unaffected by this change'. Both go through the exact ProjectedContainerProcessRunner.start() this PR changes.
    Why it's real: The only production construction of both classes is projected-container.ts:516-517 over this.process. Neither has a separate exec path. Both call this.process.start().
    Suggested fix: After adding stdin:'pipe' at both call sites, correct the body to list every stdin writer and say how each is handled.
  4. [important, fix: small] packages/agent/src/tools/runtime/__tests__/container-exec-fs.test.ts:29 and container-exec-network.test.ts fake runners; projected-container.test.ts:651-697
    No test composes the real container runner with its stdin-writing consumers, so CI stays green while container writes and POSTs are broken. The fs/network suites use fake runners whose start() ignores opts and always returns a stdin. The new tests call only runtime.process.start directly.
    Why it's real: The fake start(command) signature has no opts parameter, so it cannot observe the stdin option. Both reviewers ran the relevant suites on the PR head, and they all passed while the composed repros failed.
    Suggested fix: Make the fake runners honor opts.stdin (return stdin only for 'pipe'), or add composed ProjectedContainerToolRuntime tests for fs.writeTextFile and network.fetch with a body. Either one would have failed on this PR.

Refuted by the verifier: The 'does not end the container stdin pipe when the caller opts into stdin: pipe' test is vacuous (minor); Duplicate findings from Reviewers A and B (items 1-4 each)

https://claude.ai/code/session_01PvaV8ypCvokp6DCcW6rT7f

@ada-sen
ada-sen force-pushed the fix-pri-3250-container-runtime-stdin branch from 21eb956 to d449f51 Compare September 25, 2026 19:38
ada-sen added a commit to ada-sen/lace that referenced this pull request Sep 25, 2026
…oudly, not silently (PRI-3250)

jc's review of obra#415 (on d449f51) found the fix itself broke container
mode: RuntimeProcessRunner.start() now ends stdin by default (correctly,
per this PR), but the only two production callers that write to stdin never
opted into stdin:'pipe'.

Audited every caller of RuntimeProcessRunner start()/exec() in container
mode:
- bash.ts (foreground bash tool) — passes stdin:'ignore' explicitly
  (PRI-3243); doesn't write to stdin. Fine.
- shell-job.ts (background bash jobs) — passes stdin:'ignore' explicitly;
  doesn't write to stdin. Fine.
- runtime-stdio-transport.ts (in-container MCP server) — passes
  stdin:'pipe' explicitly and genuinely drives it. Fine, unaffected.
- container-exec-fs.ts's writeTextFile — BROKEN. No opts passed at all, so
  the runner defaulted to 'ignore' and writeTextFile always threw
  "write stream unavailable". Broke file_write/file_edit for every
  container-mode runtime.
- container-exec-network.ts's fetch — BROKEN for requests with a body. No
  stdin option passed; `if (handle.stdin)` silently skipped the body write
  when handle.stdin was undefined, so curl sent an empty POST with no
  error raised.
- container-exec-fs.ts's stat/readTextFile/mkdir/readdir and
  file_find.ts/ripgrep_search.ts — all use process.exec(), never write to
  stdin. Unaffected either way.

Fix:
- writeTextFile now passes { stdin: 'pipe' }. Kept (already correct) the
  existing throw when handle.stdin is still missing.
- fetch now passes stdin: 'pipe' only when there's a body to write
  (nothing needed for GET/no-body); added a throw when hasBody is true but
  handle.stdin is missing, matching writeTextFile's fail-loud pattern
  instead of silently sending an empty body.
- Corrected the PR body/commit-message claim that these two call sites
  have their own stdin paths "unaffected by this change" — both go
  through the exact ProjectedContainerProcessRunner.start() this PR
  changes.

Tests:
- The unit-level fakes in container-exec-fs.test.ts and
  container-exec-network.test.ts previously ignored `opts` entirely, so
  they could not observe a missing stdin:'pipe' — exactly how this
  regression passed review. Updated both to honor opts.stdin (live stdin
  only for 'pipe'), matching the real runner's behavior.
- Added composed tests to projected-container.test.ts that wire the REAL
  ProjectedContainerProcessRunner to the REAL ContainerExecFileSystem /
  ContainerExecNetworkClient (the actual production integration seam,
  not a hand-rolled fake): writeTextFile succeeds and the real content
  arrives over the container's stdin pipe; a POST body arrives intact.
  Verified: both fail on d449f51 (writeTextFile throws "write stream
  unavailable"; the POST body test observes an empty write) and pass
  after this fix.
- Strengthened the existing (jc: "vacuous, minor") "does not end the
  container stdin pipe when the caller opts into stdin: pipe" test: it
  previously only asserted `end()` wasn't called and `stdin` was
  reference-equal, both true of pre-fix code too since pre-fix always left
  the pipe live. Now also writes through the returned handle and confirms
  the same bytes arrive on the container-side pipe.

Gates: packages/agent unit suite (366 files / 3663 tests passed, 35
skipped), npm run lint, npm run typecheck — all clean.
@ada-sen

ada-sen commented Sep 25, 2026

Copy link
Copy Markdown
Contributor Author

Addressed in 261ba21:

1/2. writeTextFile / fetch never passed stdin: 'pipe': both now do (fetch only when there's a body). Also made both fail loudly — fetch now throws if hasBody but handle.stdin is missing, matching writeTextFile's existing throw, instead of silently sending an empty body.
3. PR body's false "unaffected" claim: corrected — added a full audit of every RuntimeProcessRunner.start()/exec() caller in container mode (bash.ts, shell-job.ts, runtime-stdio-transport.ts are all fine; container-exec-fs.ts/container-exec-network.ts were the two broken ones; the exec()-only callers in container-exec-fs.ts/file_find.ts/ripgrep_search.ts never write to stdin either way) to the PR description.
4. Test gap — fakes don't observe opts: updated both container-exec-fs.test.ts's and container-exec-network.test.ts's fake runners to honor opts.stdin (live stdin only for 'pipe'), and added composed tests in projected-container.test.ts that wire the real ProjectedContainerProcessRunner to the real ContainerExecFileSystem/ContainerExecNetworkClient — the actual production integration seam. Confirmed both fail on d449f51 (writeTextFile throws "write stream unavailable"; the POST body test observes an empty write with no error) and pass on 261ba21.
5 (minor, refuted). Also strengthened the "does not end the container stdin pipe when the caller opts into stdin: pipe" test your verifier flagged as vacuous — it now writes through the returned handle and confirms the bytes actually arrive on the container-side pipe, not just reference-equality/not-called assertions that were equally true pre-fix.

ada-sen and others added 3 commits September 26, 2026 05:17
…e (PRI-3250)

## Problem

obra#418 (PRI-3243) gave RuntimeProcessOptions a stdin?: 'ignore' | 'pipe'
option, defaulted to 'ignore', so a foreground bash spawn doesn't hang on
a command that unexpectedly reads stdin (e.g. head with no file args).
bash.ts passes stdin: 'ignore' explicitly. But that only reaches the
host runtime (HostProcessRunner.start() in host.ts). The container
runtime's ProjectedContainerProcessRunner.start()
(tools/runtime/projected-container.ts) is the RuntimeProcessRunner
implementation used whenever a tool executes inside a container, and it
never read opts.stdin at all: optionsFor() doesn't forward it, and
start() always calls containerManager.execStream(), which always spawns
its underlying docker/container/plane CLI with -i and a piped stdin
(see docker-container.ts's, apple-container.ts's and plane-runtime.ts's
execStream() implementations). Nothing ever wrote to or closed that
pipe, so a container-run command reading stdin because it got zero file
args would hang forever, same shape as PRI-3243, just one layer down and
unaffected by that fix. This is the container-runtime half PRI-3250
calls out as left out of obra#418's scope.

Some execStream consumers genuinely need the piped stdin (an in-container
MCP server driven interactively via RuntimeStdioClientTransport-equivalent
plumbing, or ContainerExecFileSystem/ContainerExecNetworkClient's own
low-level exec-with-stdin paths, which already write + end the pipe
correctly and are unaffected). This fix only changes the
ProjectedContainerProcessRunner.start()/exec() path that backs the bash
tool (and other RuntimeProcessRunner callers) in container mode.

## Fix

- ProjectedContainerProcessRunner.start() now ends the execStream
  handle's stdin immediately (with an error listener to swallow EPIPE)
  whenever opts.stdin !== 'pipe' (the default), mirroring
  HostProcessRunner's 'ignore' semantics without needing to change the
  three container runtimes' own execStream() signatures.
- The returned RuntimeProcessHandle.stdin is undefined in that case, so
  a caller can't be misled into writing to an already-ended stream; it's
  only the live handle when the caller opted into stdin: 'pipe'.

## Tests

Added to
packages/agent/src/tools/runtime/__tests__/projected-container.test.ts:

- 'ends the container stdin pipe immediately when the caller does not
  opt into it' — asserts end() is called once, the stream is
  writableEnded, and the returned handle.stdin is undefined. Fails
  against the pre-fix code (end() never called).
- 'does not end the container stdin pipe when the caller opts into
  stdin: pipe' — asserts the opposite for an MCP-server-shaped stdin:
  'pipe' caller.

Gates: full packages/agent test suite (3656 passed, 35 skipped, 3
pre-existing failures unrelated to this change — confirmed by running
them in isolation on this same worktree before this commit:
resource-resolver.test.ts fails because this fresh worktree has no
built dist/, and track-compaction.fixedpoint.test.ts's one failure was a
parallel-run timing flake that passes standalone), npm run lint, npm run
typecheck — all clean.
…oudly, not silently (PRI-3250)

jc's review of obra#415 (on d449f51) found the fix itself broke container
mode: RuntimeProcessRunner.start() now ends stdin by default (correctly,
per this PR), but the only two production callers that write to stdin never
opted into stdin:'pipe'.

Audited every caller of RuntimeProcessRunner start()/exec() in container
mode:
- bash.ts (foreground bash tool) — passes stdin:'ignore' explicitly
  (PRI-3243); doesn't write to stdin. Fine.
- shell-job.ts (background bash jobs) — passes stdin:'ignore' explicitly;
  doesn't write to stdin. Fine.
- runtime-stdio-transport.ts (in-container MCP server) — passes
  stdin:'pipe' explicitly and genuinely drives it. Fine, unaffected.
- container-exec-fs.ts's writeTextFile — BROKEN. No opts passed at all, so
  the runner defaulted to 'ignore' and writeTextFile always threw
  "write stream unavailable". Broke file_write/file_edit for every
  container-mode runtime.
- container-exec-network.ts's fetch — BROKEN for requests with a body. No
  stdin option passed; `if (handle.stdin)` silently skipped the body write
  when handle.stdin was undefined, so curl sent an empty POST with no
  error raised.
- container-exec-fs.ts's stat/readTextFile/mkdir/readdir and
  file_find.ts/ripgrep_search.ts — all use process.exec(), never write to
  stdin. Unaffected either way.

Fix:
- writeTextFile now passes { stdin: 'pipe' }. Kept (already correct) the
  existing throw when handle.stdin is still missing.
- fetch now passes stdin: 'pipe' only when there's a body to write
  (nothing needed for GET/no-body); added a throw when hasBody is true but
  handle.stdin is missing, matching writeTextFile's fail-loud pattern
  instead of silently sending an empty body.
- Corrected the PR body/commit-message claim that these two call sites
  have their own stdin paths "unaffected by this change" — both go
  through the exact ProjectedContainerProcessRunner.start() this PR
  changes.

Tests:
- The unit-level fakes in container-exec-fs.test.ts and
  container-exec-network.test.ts previously ignored `opts` entirely, so
  they could not observe a missing stdin:'pipe' — exactly how this
  regression passed review. Updated both to honor opts.stdin (live stdin
  only for 'pipe'), matching the real runner's behavior.
- Added composed tests to projected-container.test.ts that wire the REAL
  ProjectedContainerProcessRunner to the REAL ContainerExecFileSystem /
  ContainerExecNetworkClient (the actual production integration seam,
  not a hand-rolled fake): writeTextFile succeeds and the real content
  arrives over the container's stdin pipe; a POST body arrives intact.
  Verified: both fail on d449f51 (writeTextFile throws "write stream
  unavailable"; the POST body test observes an empty write) and pass
  after this fix.
- Strengthened the existing (jc: "vacuous, minor") "does not end the
  container stdin pipe when the caller opts into stdin: pipe" test: it
  previously only asserted `end()` wasn't called and `stdin` was
  reference-equal, both true of pre-fix code too since pre-fix always left
  the pipe live. Now also writes through the returned handle and confirms
  the same bytes arrive on the container-side pipe.

Gates: packages/agent unit suite (366 files / 3663 tests passed, 35
skipped), npm run lint, npm run typecheck — all clean.
…w narration (PRI-3250)

- container-exec-fs.test.ts / container-exec-network.test.ts: the fakes can
  now withhold stdin even for a 'pipe' caller and count kill() calls. New
  tests assert writeTextFile and a body-bearing fetch kill the process and
  throw "write stream unavailable" instead of writing nothing, and that a
  body-less fetch does not need stdin at all. Both throw tests fail if the
  guard is reverted to the old silent skip.
- projected-container.test.ts: the 'pipe' opt-in test's comment now says
  what it guards (the fix must not end or hide the pipe for 'pipe'
  callers) and that a runner ignoring opts.stdin would also pass it. The
  composed-runner describe block is renamed to say what it covers.
- Comments in container-exec-fs.ts, container-exec-network.ts,
  projected-container.ts and the tests now give the reason for the code
  and drop review-history narration and commit SHAs.
- types.ts: RuntimeProcessOptions.stdin is now documented as honored by
  every runtime, including the projected container runtime.
@obra
obra force-pushed the fix-pri-3250-container-runtime-stdin branch from 261ba21 to e81fc3a Compare September 26, 2026 05:17
@obra
obra marked this pull request as ready for review September 26, 2026 05:45
@obra
obra merged commit 62c26e4 into obra:main Sep 26, 2026
1 check passed
ada-sen added a commit to ada-sen/lace that referenced this pull request Sep 26, 2026
…l window (PRI-3243)

This PR now covers only the detached-process-group / kill-model rework.
The stdin fix shipped separately as obra#418; obra#414 (foreground timeout) and
obra#415 (container-runtime stdin) are already on main. This is rebuilt from
scratch on top of that main, not rebased.

jc's review of the previous version of this PR found the detached-group
fix incomplete: a subagent kill only gives the subagent 500ms
(job-control.ts's killJob/killAllRunningJobs, both called with
waitMs: 500, forceKill: true) before SIGKILLing it outright. The prior
attempt's shutdown()-time reap used a 2s SIGTERM->SIGKILL grace period —
comfortably past that 500ms deadline, so a TERM-ignoring descendant of a
detached foreground bash command would still be mid-poll when the
parent's SIGKILL ended the subagent process outright, abandoning the
detached group exactly as before.

Fix, same shape as before but re-timed to fit the real deadline:
- bash.ts spawns its foreground command detached (POSIX) again and kills
  the whole process group (kill(-pid)), not just the shell, on both abort
  and timeout.
- process-group-registry.ts tracks every detached command's pid;
  main.ts's shutdown() calls killAllTrackedProcessGroups() as the very
  first thing it does (before workspace teardown, before anything else
  that can hang), SIGTERMing every tracked group, polling for it to empty
  out, and SIGKILLing any survivor.
- The grace period for that escalation is now 150ms, not 2s -- far under
  the parent's 500ms deadline with room for signal-delivery and poll-loop
  latency, so the reap reliably finishes before the parent's SIGKILL can
  land on the subagent process itself.

Why this design and not the others considered:
- A pgid-file/handshake so the PARENT tracks and kills descendant groups
  itself would work too, but needs a new IPC surface between parent and
  child and still has to solve the same "how fast can this finish" problem
  -- it doesn't remove the timing constraint, just moves the deadline to a
  different process. Reaping from the subagent's own SIGTERM handler needs
  no protocol change: the subagent already gets the SIGTERM, and it already
  owns the process-group-registry that named these pids.
- Not detaching at all would lose the ability to kill a pipeline (e.g.
  sleep | cat) from a plain abort inside the SAME process, which is the
  behavior obra#414's review already validated for this PR; dropping it would
  regress that.
- A larger grace period is what jc's review actually found broken: it has
  to be short specifically because the parent's budget is short.

Tests:
- New: agent-process.shutdown-process-group.e2e.test.ts. Spawns a real
  lace-agent process, gives it a foreground bash command with BOTH a
  plain `&` background child and a foreground pipeline whose first stage
  traps (ignores) SIGTERM, then kills the agent process the same way
  job-control.ts's killJob does -- SIGTERM, wait only 500ms, SIGKILL if
  still running -- and asserts neither descendant survives.
  - Fails on plain main (no detached spawn at all: the bash-tool child is
    never signaled when the agent process exits, so it's simply
    reparented to init).
  - Fails on main + a detached-only spawn with no registry/shutdown reap
    (confirmed by temporarily reverting just the main.ts change): the
    detached group is abandoned to init the moment the agent process
    exits, since nothing ever signals a group outside the agent's own.
  - Passes with the full fix.
- New: process-group-registry.test.ts (tracking/untracking, SIGKILL
  escalation for a TERM-ignoring group within the 150ms/500ms budget, and
  no-op on an empty or already-untracked registry).
- bash.test.ts: added the process-group abort test (pipeline member killed
  on abort, not just the shell) and the no-unhandled-rejection-on-abort
  test.

Gates: packages/agent full suite (`npm run test`) -- 374 files / 3851
tests passed, 35 skipped (unit phase) + 48 files / 276 tests passed, 20
skipped (E2E phase), 0 failed. `npm run lint`, `npm run typecheck` --
both clean. No orphan processes left on the host after the suite run.
obra added a commit that referenced this pull request Oct 3, 2026
…l window (PRI-3243) (#413)

* fix(agent): reap detached bash process groups within the parent's kill window (PRI-3243)

This PR now covers only the detached-process-group / kill-model rework.
The stdin fix shipped separately as #418; #414 (foreground timeout) and
#415 (container-runtime stdin) are already on main. This is rebuilt from
scratch on top of that main, not rebased.

jc's review of the previous version of this PR found the detached-group
fix incomplete: a subagent kill only gives the subagent 500ms
(job-control.ts's killJob/killAllRunningJobs, both called with
waitMs: 500, forceKill: true) before SIGKILLing it outright. The prior
attempt's shutdown()-time reap used a 2s SIGTERM->SIGKILL grace period —
comfortably past that 500ms deadline, so a TERM-ignoring descendant of a
detached foreground bash command would still be mid-poll when the
parent's SIGKILL ended the subagent process outright, abandoning the
detached group exactly as before.

Fix, same shape as before but re-timed to fit the real deadline:
- bash.ts spawns its foreground command detached (POSIX) again and kills
  the whole process group (kill(-pid)), not just the shell, on both abort
  and timeout.
- process-group-registry.ts tracks every detached command's pid;
  main.ts's shutdown() calls killAllTrackedProcessGroups() as the very
  first thing it does (before workspace teardown, before anything else
  that can hang), SIGTERMing every tracked group, polling for it to empty
  out, and SIGKILLing any survivor.
- The grace period for that escalation is now 150ms, not 2s -- far under
  the parent's 500ms deadline with room for signal-delivery and poll-loop
  latency, so the reap reliably finishes before the parent's SIGKILL can
  land on the subagent process itself.

Why this design and not the others considered:
- A pgid-file/handshake so the PARENT tracks and kills descendant groups
  itself would work too, but needs a new IPC surface between parent and
  child and still has to solve the same "how fast can this finish" problem
  -- it doesn't remove the timing constraint, just moves the deadline to a
  different process. Reaping from the subagent's own SIGTERM handler needs
  no protocol change: the subagent already gets the SIGTERM, and it already
  owns the process-group-registry that named these pids.
- Not detaching at all would lose the ability to kill a pipeline (e.g.
  sleep | cat) from a plain abort inside the SAME process, which is the
  behavior #414's review already validated for this PR; dropping it would
  regress that.
- A larger grace period is what jc's review actually found broken: it has
  to be short specifically because the parent's budget is short.

Tests:
- New: agent-process.shutdown-process-group.e2e.test.ts. Spawns a real
  lace-agent process, gives it a foreground bash command with BOTH a
  plain `&` background child and a foreground pipeline whose first stage
  traps (ignores) SIGTERM, then kills the agent process the same way
  job-control.ts's killJob does -- SIGTERM, wait only 500ms, SIGKILL if
  still running -- and asserts neither descendant survives.
  - Fails on plain main (no detached spawn at all: the bash-tool child is
    never signaled when the agent process exits, so it's simply
    reparented to init).
  - Fails on main + a detached-only spawn with no registry/shutdown reap
    (confirmed by temporarily reverting just the main.ts change): the
    detached group is abandoned to init the moment the agent process
    exits, since nothing ever signals a group outside the agent's own.
  - Passes with the full fix.
- New: process-group-registry.test.ts (tracking/untracking, SIGKILL
  escalation for a TERM-ignoring group within the 150ms/500ms budget, and
  no-op on an empty or already-untracked registry).
- bash.test.ts: added the process-group abort test (pipeline member killed
  on abort, not just the shell) and the no-unhandled-rejection-on-abort
  test.

Gates: packages/agent full suite (`npm run test`) -- 374 files / 3851
tests passed, 35 skipped (unit phase) + 48 files / 276 tests passed, 20
skipped (E2E phase), 0 failed. `npm run lint`, `npm run typecheck` --
both clean. No orphan processes left on the host after the suite run.

* fix(agent): stop untracking a detached process group on shell completion

The detached-process-group registry untracked a pgid as soon as the one
pid it was given (the shell bash.ts spawned directly) exited, using that
process's own completion promise as the removal signal. A plain `cmd &`
backgrounds a grandchild in the same group that outlives the shell: the
shell returns almost immediately after launching it, so the entry was
dropped from the registry long before the background job was actually
done. killAllTrackedProcessGroups() then had nothing left to signal for
that survivor, so a background child of a foreground bash-tool call could
outlive the agent process entirely.

Remove the completion-driven auto-untracking. A tracked pgid is now
removed only once its process group is observed empty (groupIsEmpty),
checked lazily: opportunistically whenever a new pgid is tracked, and
during the kill sweep itself after the SIGTERM/SIGKILL escalation has had
its chance to empty the group out. A pgid with live members can't be
reused by the OS for something unrelated, so deferring removal this way
carries no pid-reuse risk, and the registry still can't grow without
bound over the life of a long-running agent process.

Adds a regression test that spawns `sleep 30 &` through the real bash
tool, lets the shell exit, runs the kill sweep, and asserts the
background process is dead -- it fails against the prior behavior and
passes with this fix. Also updates the existing process-group-registry
unit tests for the new (no-completion-arg) trackProcessGroup signature
and adds coverage for the lazy-removal behavior directly.

Does not change the detached-process-group's own reachability from a
force-killed parent (a separate, still-open design question); see the PR
description for a follow-up note on that trade-off.

* fix(agent): spawn the shutdown-process-group E2E test agent detached (jc finding #4)

jc's finding #4 on 20a4539: the E2E test's killTheWayTheParentDoes called
plain proc.kill() on a non-detached spawnAgentProcess child, but a real
subagent is spawned detached (subagent-spawn.ts:58) and killed via
job-control.ts's real group kill (process.kill(-pid, ...)). The mismatch let
the test pass for the wrong reason and hid finding #1 (the untrack-on-
completion regression, fixed in 0cfa772): with a foreground pipeline that
blocks until killed, the shell's completion promise never resolves early
enough for the bug to matter either way.

Fix:
- spawnAgentProcess (test helper) gains a `detached` option, spawning the
  agent the same way subagent-spawn.ts spawns a real subagent.
- The E2E test now spawns detached and kills with the real job-control.ts
  killJob (wrapped in a minimal JobState), not a hand-rolled approximation.
- Both orphan shapes are now backgrounded with `&` (redirecting their own
  stdout/stderr so they don't hold the agent's output pipe open) so the bash
  tool call itself returns before the kill -- the shape finding #1 actually
  needs: a background job left by a call that has already returned.
- Corrected the stale "subagents are not spawned detached" claim in
  process-group-registry.ts's module doc and job-control.ts's group-kill
  fallback comment; both predated subagent-spawn.ts's detached spawn and
  were flagged by jc as the likely source of the test's false premise.

TDD: copied the corrected test (+ helper) into a scratch worktree at
20a4539 (the commit immediately before the untrack-on-completion fix) and
confirmed it fails there -- `expect(isAlive(trapPid)).toBe(false)` gets
`true` (orphan survives), reproducing finding #1 under the now-realistic
detached+real-killJob methodology. Reapplied on top of 0cfa772 (this
branch) plus this change: passes, 5/5 runs, no orphan processes left behind
either way (confirmed and cleaned up manually after the red runs).

Does not touch the reg2 design question (parent SIGKILL reachability into
the bash tool's own detached group) -- split out per jc's finding #2.

Full packages/agent suite: 422 files / 4131 tests passed, 0 failed (55
skipped, pre-existing). ent-protocol and cli suites green. Lint, typecheck
(whole monorepo), and prettier --check all clean.

* fix(agent): kill groups tracked after the shutdown reap, release emptied groups on shell exit

The shutdown reap is a one-time snapshot and shutdown() does not abort
the running turn, so a bash call starting after the reap had its group
tracked but never reaped. The first killAllTrackedProcessGroups call now
puts the registry into a shutting-down state, in which
trackProcessGroup SIGKILLs the new group at once.

A tracked pgid also lingered after its group emptied until the next
lazy sweep, which let a recycled pid that became an unrelated group
leader be signaled at shutdown. bash.ts now releases its group when the
shell exits and the group is observed empty.

The registry module doc now covers stale-entry pid reuse, that
background jobs started by bash calls do not survive an agent restart,
the shutting-down state, and the known gap where an unresponsive
subagent's groups are orphaned by the parent's SIGKILL (#419).

* test(agent): cover in-flight and post-cancel kills in the shutdown process-group E2E

Adds two cases against a real detached agent killed with the real
killJob: a TERM-ignoring pipeline killed while its bash call is still
in flight, and the same pipeline killed ~1500ms after session/cancel,
inside the bash tool's own 2s SIGKILL grace. Every recorded descendant
pid is SIGKILLed in afterEach so a failing run cannot leak sleeps to
init. Rewords the ABOUTME to describe what the file covers.

---------

Co-authored-by: Jesse Vincent <jesse@fsck.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants