Detached Harper children are orphaned permanently when the runner dies by SIGKILL/SIGHUP — reap guard only covers exit/SIGINT/SIGTERM

[kriszyp]

Summary

trackHarperProcess reaps spawned Harper children on exit, SIGINT, and SIGTERM. It does not cover the ways a test runner most often actually dies — SIGKILL, SIGHUP, or an abrupt crash — and because Harper is spawned detached (its own process group) the child does not die with its parent. The result is a Harper instance that survives indefinitely with PPID 1, still holding its listening ports.

Evidence (observed on a dev machine, 2026-08-20)

Nine orphaned dist/bin/harper.js processes, all PPID 1, all cwd'd in one harper-pro worktree, ages 16–18 hours:

  PID  PPID  %CPU     ELAPSED
93582     1  99.6    18:15:17
94348     1  99.6    18:13:33
29069     1  98.7    16:39:01
28383     1  98.7    16:40:41
28458     1  98.5    16:40:26
 2574     1  98.4    18:00:40
93587     1  98.1    18:15:13
 3839     1  97.9    17:59:05
 2577     1  96.3    18:00:35

Combined 879% CPU (~8.8 cores). Machine load average was 21.8 with 0.14% idle; killing these nine plus two unrelated runaway jobs took it to 6.9 / 91% idle.

They were still holding their ports, which is the part that breaks subsequent runs:

node 93582 ... TCP 127.0.0.7:8883 (LISTEN)
node 93582 ... TCP 127.0.0.7:1883 (LISTEN)
node 93582 ... TCP 127.0.0.7:9933 (LISTEN)
node  2574 ... TCP 127.0.0.10:9933 (LISTEN)
node 28383 ... TCP 127.0.0.15:9933 (LISTEN)

A separate 18-day-old orphaned mocha runner was also present on the same box, so this is not a one-off.

All nine exited cleanly on a plain SIGTERM — nothing was wedged at the OS level; no one had signalled them.

Why the current guard misses

src/harperLifecycle.ts (trackHarperProcess, ~L704 on main):

process.once('exit', reapAll);
// SIGINT/SIGTERM don't fire 'exit'; reap, then re-raise so the runner still terminates normally.
for (const signal of ['SIGINT', 'SIGTERM'] as const) {
    process.once(signal, () => {
        reapAll();
        process.kill(process.pid, signal);
    });
}

Not covered:

Runner death	Handler runs?	Why it matters
SIGKILL	No — uncatchable	CI job cancellation, OOM killer, kill -9
SIGHUP	No — not registered	terminal/session teardown
hard crash / SIGABRT	No	runner segfault or abort
SIGINT / SIGTERM / normal exit	Yes	already handled

The detached: true spawn is correct and deliberate (it's what lets teardown signal the whole group), but it also means the child is insulated from anything that kills the runner's group. Nothing in the child notices its parent is gone.

CI cancellation is the case worth calling out: cancelled jobs are a routine occurrence in the harper-pro stress matrix, and cancellation lands as SIGKILL.

The invariant

"A Harper child never outlives its runner" is currently maintained by enumerating parent-side signals — so every uncatchable or unregistered death is a hole. It would be more robust to enforce it from the child side, where no parent cooperation is needed.

Suggested direction (not prescriptive)

1. Parent-liveness watchdog in the child — the durable fix. Have the child poll for getppid() === 1 (or watch the parent via a pipe that closes on parent death) and self-terminate. Survives SIGKILL because it needs nothing from the parent.
2. Register SIGHUP alongside SIGINT/SIGTERM — cheap, closes one hole, doesn't help with SIGKILL.
3. Startup sweep — on startHarper, reap stale Harper processes whose parent is gone before claiming an address. Complements #13, which is the same runner-killed-mid-shard root cause applied to the loopback address pool.

(1) is the only one that closes the SIGKILL case; (3) is the pragmatic backstop and would also reduce the port-contention failures behind #8 and #28.

Related

• #13 — Loopback pool leaks addresses from runners killed mid-shard (dead-PID sweep only runs when the pool is full) — same root cause (runner killed mid-shard), different leaked resource.
• #8 — startHarper readiness deadline and teardown loopback recycle cause intermittent failures under sharded CI and #28 — Teardown's port-release wait probes all five fixed Harper ports unconditionally — orphans holding ports are one source of the contention these describe.

Separately: the reason these orphans were hot rather than idle is a Harper-side unbounded spin lock, filed separately against harper. Reaping and the spin are independent defects that compounded here.

Version

@harperfast/integration-testing 0.7.1; guard is unchanged on main.