Runner execution-policy guidance

[lann]

Collected policy patterns for L3 runners, mostly from the webrtc-datachannels conformance suite (patterns that survived the webcrypto fork are additionally marked ✓):

• Instance granularity is runner policy. Fresh-instance-per-case gives isolation, trivial trap containment, and parallelism via replication (intra-instance concurrency is cooperative-only — replication is the parallelism axis); instance-per-suite is cheaper but poisoning costs the remainder (recover by re-instantiating + resuming by name). Both legal under L1. ✓ (both systems are fresh-per-case/process)
• Layered timeouts: runner hang guard > any SUT-internal operation timeout, so genuine failures classify as failed outcomes and only true hangs trip the guard (guard trip = trap-equivalent = abandon instance). Phase markers in diagnostics identify the hung phase.
• No retries ✓: a flaky bug would be masked by a second attempt; for network nondeterminism prefer deterministic simulation environments over retries.
• Write-through reporting: forward diagnostics to the transport promptly; anything buffered in a wasm runner core dies with the store on a trap (prototype finding).
• Cost tiers / corpus subsetting are deselected, not not-applicable: restricting an expensive target (e.g. browser-boot-per-case) to a smoke subset is selection policy, visible as such in results — never conflated with capability.
• Concurrent drain obligation: a runner observing diagnostics must consume the stream concurrently with run (load-bearing for sync-lifted suites too — they wedge, not trap, under a non-draining observer).
• Parallelism multipliers need empirical tuning per target class (browser targets saturate at lower multiples; load-induced timeouts are the flake signature).

Destination: runner-implementer documentation + defaults in the reference runners; the results schema work inherits the deselected/not-applicable distinction.

[lann]

Execution-default update from further discussion:

Instance-per-case should be the reference default (at minimum for wasmtime runners, where the pooling allocator makes instantiation ~microseconds). Rationale:

• deletes machinery: no resume protocol, no case-trap vs resume-trap distinction — a trap kills one case's store, the runner records a trap-provenance fail and instantiates the next;
• self-containment becomes mechanical fact rather than discipline (no shared state exists to leak), and the insecure-seed nuance from the wasi:random work vanishes;
• parallelism is replication with sharding granularity 1.

Costs and mitigations:

• Enumeration cost goes O(N²) (each instance runs all() for one case): mitigate with lazy case materialization (SDK guidance), pre-initialized suites (Pre-initialized suites (build-time wizer snapshot of the case table) #25) which amortize all enumeration to build time, and — if measured to matter — an additive suite-side direct-access interface or Future idea: cases-as-exports producer format (via implements annotations) #18 cases-as-exports.
• Granularity must be a per-target-class knob, not universal: per-case startup dominates on Node/Chromium targets (the webrtc-datachannels suite's flake history is exactly load-induced per-case boot timeouts) — jco-browser wants per-chunk or per-suite instances.

Shared-instance mode remains the documented fallback (cheap targets, expensive instantiation), and inherits the trap-recovery machinery from the earlier discussion: resume-by-index against the lockfile, harness-error vs case-trap distinction (trap before reaching the resume index = harness error, preventing re-instantiation loops), segmented result assembly, diagnostics-are-prefix on trap verdicts, and selection/resume promoted into the CLI's core interface.

[lann]

Two production refinements to the replication doctrine, from webcrypto's cc61973 (worker-pool jco adapter — instance replication, strictly sequential per instance, exactly this issue's model): (1) modulo striping (index % workers) over contiguous chunks, because expensive cases cluster by algorithm in generated corpora; (2) deterministic re-sort before the wire — workers emit as-completed, parent restores census order, so parallel execution yields byte-stable result files. For us the re-sort belongs at the document fold.

[lann]

The granularity knob is now implemented and measured (commit b3c9eb7 + the webcrypto branch). Profiling the 11,578-case corpus under instance-per-case: 95% of wall time was all() (per-instance registry build + lifting 11.5k handles, 6.9ms/case); instantiation was already 64µs (CoW images; adding the pooling allocator made no measurable difference), actual case work 358µs, teardown 57µs.

Cost ladder on the same corpus, each mechanism removing its own layer:

change	full run
baseline (JSON corpus, instance-per-case)	11:36
build-time preprocessing (postcard)	3:35
zero-copy corpus (rkyv access_unchecked)	1:23
single-instance topology (cases_per_instance = 0)	4.2s
incumbent adapter (single instance, concurrent)	4.3s

Doctrine update this data forces: instance-per-case remains the correct default (isolation, trap containment — and a trap abandons the instance regardless of the knob), but for cheap pure-compute corpora it is a 20–160× tax buying nothing; runners for such targets should tune K high or single-instance. The remaining delta to beat the incumbent (their case-driving is concurrent within the instance) is #20's territory.

Also relevant to #25: with rkyv + single-instance, the wizer motivation on this corpus is now fully absorbed (registry builds once in 12ms); #25 stays filed for suites that must keep K small.

[lann]

The parallelism knob is implemented (0e4f23e): --jobs N — scoped worker threads sharing the compiled engine, owning their stores, running modulo stripes (runnable-index % jobs == worker), results emitted in census order (byte-stable output regardless of completion order). cases_per_instance applies per worker; K=0 under parallelism means instance-per-worker (the replication doctrine, literally).

Final campaign numbers on the webcrypto corpus (11,578 cases, 17-core):

config	wall
campaign start (JSON corpus, K=1, sequential)	11:36
K=1 sequential (all optimizations)	36s
K=1, jobs=8 — full per-case isolation	7.3s
K=0, jobs=8 — instance-per-worker	2.3s
incumbent adapter (single instance, in-instance concurrency)	4.3s

Full isolation now costs <2× the incumbent's non-isolated run; the replication mode beats the incumbent outright without any intra-instance concurrency (#20 remains unexercised). The "instance-per-case is a 20–160× tax" caveat earlier in this thread is hereby amended: with a preprocessed corpus and striping, it's a ~1.7× tax against the fastest mode — the default-vs-tuned framing stands, but the default is no longer painful at scale.