Process-isolate storyboard execution and enforce per-storyboard resource budgets

[bokelley]

Problem

The current storyboard CI runner loads the training agent, SDK compliance runtime, compiled schema graph, embedded HTTP server, and multiple storyboards into one Node.js process. Reset helpers clear known logical stores between storyboards, but they cannot prove that every SDK cache, validator graph, timer, transport, socket, and module-level object has been released.

This allowed latent correctness bugs to turn into runner-wide resource failures:

• webhook_emission: no result after ~97 seconds, ~13.7 GB peak RSS (adcp-client#2653).
• wholesale_feed_products_scope_isolation: incorrect public/account projection, ~8.3 GB peak RSS (adcp-client#2654).
• Adjacent proposal-finalization cases: committed-state handoff failure, ~17.6 GB combined peak RSS (adcp#6776).

The hosted runner then terminates the shard before it emits totals. Increasing the matrix to 48 fresh hosted jobs and quarantining the pathological storyboards restored CI, but it is expensive containment rather than a durable runner architecture.

Scope

This issue owns runner infrastructure only:

1. Execute each storyboard in a fresh child process. The orchestrator MUST NOT load the training agent or SDK schema graph into its own long-lived process.
2. Apply a per-storyboard wall-clock limit and a calibrated memory/RSS budget.
3. Terminate the complete child process group on timeout, memory breach, or completion.
4. Record structured per-storyboard telemetry: elapsed time, peak RSS, exit code/signal, timeout/resource-limit status, and whether a complete result envelope was emitted.
5. Aggregate child results without losing successful siblings when one storyboard crashes or is killed.
6. Keep the required lane on the standard GitHub-hosted runner. A larger runner MAY be used only for non-required diagnostic heap profiling.
7. Once child isolation is proven, reduce the current 48 checkout/install jobs to a smaller fixed matrix (target 4–8 orchestrators), each running assigned storyboards as isolated children.

This issue does not own the underlying behavioral fixes. Those remain in #6776, adcontextprotocol/adcp-client#2653, and adcontextprotocol/adcp-client#2654.

Design constraints

• Do not use worker threads; they share the process and do not provide a clean heap/module boundary.
• Do not depend on graceful SDK/V8 disposal to make progress after a storyboard has emitted its result.
• Resource limits must be calibrated from measured healthy-storyboard baselines rather than arbitrary thresholds.
• Current and released compliance bundles must retain their existing applicability semantics.
• A killed child must produce a precise runner diagnostic instead of a missing-totals ambiguity.
• Logs and telemetry must remain synthetic-test-safe and suitable for CI artifacts.

Acceptance criteria

• A test fixture that intentionally leaks memory or hangs is isolated, killed, and reported without killing the orchestrator or losing sibling results.
• A child that emits a valid result and then hangs during disposal is terminated after its result is durably captured.
• Peak RSS and elapsed time are visible per storyboard in CI artifacts.
• Repeated execution of the full current /sales set has bounded orchestrator memory.
• The current /sales required checks pass on the standard hosted runner with a substantially smaller job matrix.
• Existing required-clean, exact-result, and aggregate floor checks remain fail-closed.
• Temporary quarantines are removed only when their owning behavioral issues are fixed; process isolation must not turn a failing storyboard into a passing one.

Follow-up ownership

If profiling identifies retained objects inside @adcp/sdk/testing, open a focused adcp-client issue with heap evidence and an explicit disposal/reset contract. Do not pre-assign unproven retention to the SDK.

[bokelley]

Triage

Classification: Bug (runner-wide resource failure — CI termination from shared-process heap accumulation)
Bucket(s): compliance-suite (storyboard runner infrastructure)
Status: ready-to-implement

What the experts said:

• ad-tech-protocol-expert: Sound engineering approach — child_process isolation is the correct Node.js primitive for clean heap/module boundary separation; no normative protocol semantics are affected; applicability semantics of released compliance bundles are preserved by design constraint; acceptance criteria cover the key conformance-safety properties (fail-closed pass/fail gate, quarantine policy, sibling survival).
• code-reviewer: child_process.fork() / spawn() is the right primitive over worker threads (which share the V8 heap and module cache); primary implementation risks are IPC reliability (result envelope must be captured before SIGKILL), process group teardown on Linux (use detached: true + process.kill(-pid, 'SIGKILL') to kill the entire group), and RSS polling interval (short enough to catch a leak, long enough not to bias latency); acceptance criteria are testable and the change is runner-infra-only with no storyboard pass/fail semantic change.

My take: This is a well-scoped runner infrastructure fix — the 48-job containment workaround is expensive, and the underlying architecture (single shared process) cannot prove isolation by design. The implementation path is clear: orchestrator that forks one child per storyboard, monitors RSS + wall-clock, SIGKILLs the process group on breach, captures the result envelope before teardown via IPC, and aggregates results. No protocol or spec change; no changeset needed.

Implementation scope:

• Add an orchestrator module (likely alongside server/tests/manual/run-storyboards.ts) that forks one child per storyboard using child_process.fork() with detached: true; pass the storyboard ID as an arg.
• Each child: load the training agent + SDK runtime, run exactly one storyboard, emit the result envelope over IPC, then exit. The orchestrator captures the envelope before it terminates the child.
• Resource monitor: poll process.memoryUsage().rss (or /proc/{pid}/status VmRSS) on a short interval (e.g., 500ms); calibrate the budget from the measured healthy-storyboard baseline (the issue rightly requires measured baselines, not arbitrary thresholds).
• Timeout: setTimeout(() => process.kill(-pid, 'SIGKILL'), wallClockLimit) on the orchestrator side; clear it when the child exits normally.
• Telemetry: emit a structured JSON record per storyboard — { storyboard_id, elapsed_ms, peak_rss_bytes, exit_code, signal, timed_out, resource_limited, result_received } — as a CI artifact.
• Matrix reduction (48→4–8 orchestrators): follow-up once isolation is proven; the quarantine removals (webhook_emission, wholesale_feed_products_scope_isolation, the proposal-finalization cases) are gated on their owning behavioral issues (Training storyboard proposal finalization loses committed state #6776, adcp-client#2653, adcp-client#2654) and must not be removed as part of this PR.
• No changeset — this is CI runner infrastructure, not a protocol surface change.

---

Triaged by Claude Code. Session: https://claude.ai/code/session_01Mcm9ovjPFtoKZRdfmb5zv5

---

Generated by Claude Code