fix(agent): enforce a foreground bash timeout (PRI-3251)

[ada-sen]

Based directly on main. It no longer depends on #413. The stdin half of #413 merged as #418. The process-group half (detached spawn, process-group registry, shutdown reap) is deferred to #413 and is not in this PR.

Problem

The bash tool description has long said a sync command "is subject to a runtime timeout (tens of seconds) and is killed if it exceeds it". Nothing enforced that. A hung foreground command blocked until the caller aborted. (PRI-3251)

Fix

• Default timeout: 600s. It is configurable per instance with LACE_BASH_FOREGROUND_TIMEOUT_MS, which can only lower it. The value is clamped to 1000..600000. A value that isn't a plain positive decimal integer (10s, 0, 1e3, 0x3e8, 1000.0) falls back to 600s. Surrounding whitespace is ignored. Both cases log a warning.
• Per-call override: a new timeoutMs field, bounded 1000..600000.
• Behaviour change to note: any foreground call that omits timeoutMs and runs longer than 600s (or the lowered instance default) is now killed. Before this PR it ran indefinitely.
• On expiry: SIGTERM the shell the tool spawned, then SIGKILL it 2s later. On main the foreground child is not detached and has no process group of its own, so only the direct child is signaled. The SIGKILL timer is never cleared when the call settles. The abort path uses the same code, so a TERM-ignoring shell still dies after an abort settles the call early.
• Bounded by timeoutMs + 2s: once the SIGKILLed shell exits, the tool stops waiting on stdout/stderr. It destroys the streams and completes with the output captured so far, so a descendant holding the pipes can't keep the call open. Destroying the streams also closes the tool's read ends, so that descendant's later writes fail with EPIPE instead of landing in a settled call.
• Shell already exited, pipes still open: this happens when a background child holds stdout/stderr. The result reports the shell's real exit code, and the message says the pipes stayed open and suggests background=true or redirecting that process's output. It does not say "killed", and the tool ends the call at the timeout with no grace wait. To tell this case apart, the host runtime now exposes an optional exited promise on process handles. It resolves when the process exits, before its pipes close. Only the host runner implements exited, so under the projected container runtime this pipes-held message can't fire, and such a call gets the regular killed-shell message instead.
• Result shape: timedOut: boolean, plus timeoutMessage when it is true. A timed-out call is always a tool failure. When the shell was killed, the message says that processes it started may still be running. On main that is true, because only the shell is signaled.

Tests

All tests use real processes. Pid-file tests write $$ and count a process as dead only on ESRCH. They are in packages/agent/src/tools/bash.test.ts:

• The exact tool description with the env var unset. This pins the 600000 default.
• Schema rejects timeoutMs 999 and 600001 and accepts 1000 and 600000.
• Env values: unset gives 600000 (no warning). 120000 is used as given (no warning). 1800000 becomes 600000 and 500 becomes 1000 (clamped, with a warning). 10s, 0, 0x3e8, 1e3, 1000.0 and the empty string become 600000 (with a warning). 120000\n is accepted without a warning. The configured default applies to a call that omits timeoutMs.
• A hung command times out at 1s and keeps its partial output.
• A TERM-ignoring command (trap '' TERM; ... exec sleep 60) on the timeout path ends between timeout+grace-100ms and timeout+grace+1.5s, and its pid is gone.
• A setsid-escaped descendant holding the pipes doesn't hold the call past timeout+grace.
• After the call settles, a setsid-escaped descendant's late write to stdout fails (non-zero status), because the tool closed its read ends. Cleanup kills that descendant's process group.
• A shell that exited 0 while a background child holds the pipes gets exitCode: 0 and the pipes-held message, ending before timeout + 1.5s.
• A fast command is not flagged.
• A TERM-ignoring command is SIGKILLed after an abort settles the call.

Mutation checks: removing the SIGKILL fails both TERM-ignoring tests. Removing the stop-waiting step fails the escaped-descendant test. Removing both destroy() calls in settleWithoutWaitingForPipes fails the late-write test (the write succeeds, status 0). The settle bound itself doesn't depend on destroy(), since that function marks the streams ended directly, so the escaped-descendant timing test still passes without it.

Changes in this round

• Merged current main (fix(agent): honor RuntimeProcessOptions.stdin in the container runtime (PRI-3250) #415, container stdin handling). It merged cleanly.
• Added the late-write test for the stream destroy() and tightened env parsing to plain decimal integers.
• Rebuilt on current main. All fix(agent): reap detached bash process groups within the parent's kill window (PRI-3243) #413 commits are dropped, including the .finally process-group untrack line that crashed vitest.
• The timeout now kills the direct child instead of a process group.
• Env var: documented as lower-only, with warnings on clamping or unparseable values.
• New pipes-held message.
• New tests: the 600s default, schema bounds, env mappings, and a TERM-ignoring command on the timeout path.
• The pipeline-grandchild test is dropped. It tested fix(agent): reap detached bash process groups within the parent's kill window (PRI-3243) #413's process-group kill, which isn't on main.

Gates

npm run build, lint, typecheck and format:check pass. bash.test.ts has 57 tests, all passing. The full agent suite passes (3853 passed, 27 skipped; serial group 283 passed, 12 skipped). The cli and ent-protocol suites pass. test:coverage passes except track-compaction.fixedpoint.test.ts, which hits its 5s timeout on my loaded local machine. It fails the same way on untouched main and is green in main's CI.

Not in scope

• Killing descendants of a timed-out command. That needs fix(agent): reap detached bash process groups within the parent's kill window (PRI-3243) #413's process groups.
• Background (background: true) jobs. They are routed to the job system before they reach this tool.

[obra]

Review from jc: two competing adversarial reviewers, then a verifier that tried to refute each finding. Verdict: changes needed.

1. Important: settling clears the SIGTERM→SIGKILL grace timer. (clearPendingTimers, bash.ts:198/:282 with :336-342)
   • If the tool settles inside the 2s window, SIGKILL is never sent. That regresses fix(agent): reap detached bash process groups within the parent's kill window (PRI-3243) #413's abort path, where the SIGKILL timeout was never cancelled.
   • Reproduced on 68a730c:
     • Abort of trap '' TERM; sleep 47.71 returned "aborted", and the sleep was still alive 4s later.
     • The timeout path with a TERM-ignoring sh -c left both processes alive.
   • Fix: on settle, clear only the timeout timer. Let the grace timer fire (ESRCH is caught), or unref() it.
2. Important: same vacuous pid tests as fix(agent): reap detached bash process groups within the parent's kill window (PRI-3243) #413. (bash.test.ts:825, :893)
   • echo $ writes $, so the orphan assertions can't fail. Use echo \$\$, Number.isInteger, and treat only ESRCH as dead.
3. Important: the timeout doesn't actually bound the call.
   • Settlement waits for stdout/stderr end. A descendant that escaped the group (setsid, a double-forking daemon) and still holds the pipes keeps the call blocked forever.
   • Fix: after SIGKILL, destroy the streams and complete, so the call settles by timeoutMs + KILL_GRACE_MS. Add a setsid test.
4. Decision needed: the new 60s default hard-kills every foreground call that omits timeoutMs.
   • Builds, installs, clones and test runs over 60s will now fail mid-run.
   • I'm asking Jesse. Options: a higher default (e.g. the 600s max) or a configurable one, called out in release notes.
5. Minor: test gaps.
   • No test covers SIGKILL escalation (the pipeline test uses exec sleep, which dies on TERM), the default, or the schema bounds (999 / 600_001).
6. Minor: misleading timeout message.
   • When bash exits 0 but a background child holds stdout, the call waits the whole timeout and says "Command timed out" with exit 0.
   • Suggest a distinct message pointing at background=true or redirecting output.

Refuted: an unbounded container exec before the timer arms, a container stdin gap (that's #413's), and an abort-during-grace message detail.

[obra]

Jesse's decision on the default: make it configurable (a per-instance setting), with a 600s default. Please call out the behaviour change in the PR description. https://claude.ai/code/session_01PvaV8ypCvokp6DCcW6rT7f

[ada-sen]

Addressed in 8c987b9f4 (rebased onto the reworked #413, now a37884d on top of 9cc0a9f):

1. Settling cleared the SIGKILL grace timer: split clearPendingTimers — only the foreground-timeout-detection timer is cancelled on settle now; the SIGKILL grace timer runs to completion (unref'd, ESRCH caught). Reproduced first: abort of trap '' TERM; sleep 30 settles in ~1ms (Node's own spawn({signal}) abort handling fires independent of the real process) and the trap survivor is alive 3s later on 68a730c; dead (ESRCH) on the fix.
2. Vacuous pid tests: same $$/Number.isInteger/ESRCH-only fix as fix(agent): reap detached bash process groups within the parent's kill window (PRI-3243) #413, at both call sites.
3. Timeout didn't bound an escaped descendant: after the SIGKILL grace period, streams are now force-destroyed and the call settles regardless. New setsid test: (setsid sleep 30 &) ; sleep 30 with timeoutMs: 1000 times out the 15s test on 68a730c, settles in ~2s on the fix.
4. Default: made it a per-instance setting, LACE_BASH_FOREGROUND_TIMEOUT_MS, default 600s (same pattern as delegate.ts's LACE_WORKSPACE_MAX_PER_PARENT), called out in the tool description. Per-call bounds (1s–600s) unchanged.

Every new/changed test verified to fail on 68a730c and pass on 8c987b9f4.