fix(fastpath+emit): one interpreter for the test floor, and honour its verdict - #13
Merged
Merged
Conversation
…ord its output
Two defects in the tier-1 test floor, both found while dispatching a real run.
1. SHELL SYNTAX WAS EXECUTED AS ARGV. `shlex.split` tokenised the command and
`_run_supervised` exec'd the token list, so every shell construct survived as a
literal argument:
manifest: node --test $(ls test/*.test.js) && swiftc … && …
argv: ['node','--test','$(ls','test/*.test.js)','&&','swiftc', …]
result: rc=1
No command substitution, no glob, no `&&` chaining. A floor that needs any of
them fails, and the receipt makes it look like the project's tests are failing.
A string carrying POSIX-sh metacharacters now runs through `/bin/sh -c`; a
plain-argv string keeps the argv path unchanged, so existing manifests
(`npm test`, `sh -c 'exit 0'`) behave exactly as before.
2. THE RECEIPT DISCARDED THE OUTPUT. `_run_supervised` returns `(rc, stdout)` and
the caller wrote `rc, _ = …`, so a failing floor recorded only
`tier-1: configured tests failed (rc=1): <cmd>` — nothing to act on, and
indistinguishable from a genuine test failure. The bounded stdout tail is now
kept on the check and in the reason. (The supervisor discards stderr; the tail
is documented as stdout-only rather than claiming full output.)
Selftest: 8 new rows. Three are discriminators that produced the OPPOSITE verdict
before this change — `sh -c 'exit 0' && sh -c 'exit 1'` passed the floor (argv ran
only the first command) and `sh -c 'exit 1' || sh -c 'exit 0'` blocked it.
…redact Review of the previous commit found that putting the output tail into `reasons` opened a real path from a test's stdout to the next run's shell: reasons[] -> compound-v-emit-workflow.py copies any reason matching "timeout after" into tests.failures[] -> previously_failing() reads it back -> resolve_test_commands() re-runs those strings -> and a configured command now goes through /bin/sh. That copy branch was effectively dead before (tier-1 reasons say "; timeout", never "timeout after"); appending free-form command output made it reachable with attacker-influenced content. The tail now lives ONLY on `checks[].output_tail`, and the reason is a single line again. Also from review: - REDACT. The floor doc reaches receipts/<id>.gate.json, which the wave finalizer `git add`s, so the tail is durable and committed. It is now passed through the canonical PEM_RE/SECRET_RE families imported from compound-v-memory.py (the repo forbids a second copy of those patterns). If that import fails the text is dropped, not stored raw: no diagnostic is worth committing a credential. - CAPTURE STDERR. A broken command (sh syntax error, module-not-found, `ls: cannot access`) prints on stderr and nothing on stdout, so the motivating "broken command vs broken test" case was still just `rc=N`. The supervisor already supports --stderr; the floor now asks for it. Off by default so `_git` and friends keep parseable stdout. - HONEST DOCSTRING. The supervisor keeps the HEAD of each stream up to 256 KiB, so this is the tail of the first 256 KiB, not the tail of the run. Said plainly. - `shlex.split` raises on an unbalanced quote; that now fails closed with a reason instead of a traceback. - `check["via"]` records `/bin/sh -c`, which is what actually ran. Selftest: 4 new rows (reason carries no output; redaction; stderr-only failure produces a tail; unbalanced quote blocks). The output-tail fixture now prints a marker that does NOT appear in the command text, so the "no output in reasons" assertion cannot pass by accident.
…ng output
Two changes to what a `test_contract` command means and what a failing floor
records, plus the consumer half in the emitter.
1. ONE INTERPRETER. The plugin runs the SAME `resolved_commands[]` list under two
different interpreters. Every external worker runs each command through a shell,
byte-identically:
compound-v-run-codex-worker.sh:189 -- /bin/bash -c "$_tc_c"
cursor:203, antigravity:196, opencode:237
while the claude backend's tier-1 floor ran `shlex.split(raw)` as argv. So a
floor with a glob, `$(...)` or an `&&` chain PASSES on a codex job and FAILS on a
claude job of the same manifest. `floor_command` is documented only as "string"
(execution-manifest.md) and the validator checks only its type, so neither
spelling is wrong - the two backends simply disagree. argv is the odd one out,
and no commit or doc ever chose it: `shlex.split` arrived unremarked in 65114b0.
Now every command goes through the same `/bin/bash -c` the workers use. An
earlier revision shelled only strings containing metacharacters; that was worse -
it added a THIRD semantics, so `pytest tests/{a,b}` still meant one thing on
codex and another here.
`{path}` substitution is now `shlex.quote`d, matching what `referencing_tests`
already does: paths come from `git diff` of the worker's tree and are no longer
inert once every command reaches a shell.
2. THE RECEIPT KEEPS THE EVIDENCE. `_run_supervised` returns `(rc, stdout_bytes)`
and the caller wrote `rc, _ = ...`, so a failing floor recorded a bare exit code
while holding the output that explains it. The bounded tail now lands on
`checks[].output_tail`, with stderr captured too - a broken command (sh syntax
error, module-not-found) prints there and nowhere else, which is the exact
"broken command vs broken test" ambiguity this is for.
It is redacted through the canonical families imported from compound-v-memory.py,
and the docstring states the honest scope: PEM/`sk-`/`ghp_`/`AKIA`/`xox*` are
covered, `password=`, credentialed URLs and bare JWTs are NOT. This reduces what
a committed receipt can leak; it does not eliminate it.
The tail never enters `reasons[]`: the emitter copies a reason matching
"timeout after" into `tests.failures[]`, which `previously_failing()` reads and
`resolve_test_commands` re-runs - command output there would be a path from a
test's stdout to the next run's shell.
3. THE CONSUMER HALF (emitter). `run_test_floor` initialises every result
`{"passed": False, "merge_blocked": True}` and ADR 0003 relies on it: "already
merge-blocking and already fail-closed on an empty command", Consequences: "A job
reporting no test command at all is a FAIL, not a pass." `build_review_spec`
gate 1 honours that field; `_job_result_from` did not - it reads the floor only
through `_tests_block_from_floor`, which returns None when no check carries a
`checker`, so `merge_blocked: True` was never consulted and the job kept the
scope verdict's `success`. Same document, two consumers, opposite conclusions.
Dogfood 14 moved the RED half of that rule into this function; this is the empty
half. The predicate is the MANIFEST's declaration, not the resolved slice, which
is absent exactly when resolution failed. `tests is None` is `error` (broken
machinery), not `blocked`, and the refusal carries the floor's own reasons.
Unchanged: an uncontracted manifest still records `success` with no `tests`
object - "no floor at all is not a failure".
Selftests: fastpath PASSED, emitter 532/532 (was 526). Mutation-checked on both
halves. Frequency of the emitter fail-open, stated honestly: never observed - 0 of
198 recorded results here and 0 of 17 in the reporting project.
This was referenced Sep 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes defect 1 of #12 and defect 3a of #15. (#16 folded in here - it keys on the same floor document and should be bisected with this.)
Two halves of one story: what a
test_contractcommand means to its executor, and what the floor document means to its consumer.1. One interpreter for one command list
The plugin runs the same
resolved_commands[]under two different interpreters. Every external worker runs each command through a shell, byte-identically:while the claude backend's tier-1 floor ran
shlex.split(raw)as argv (compound-v-fastpath-run.py:1167,1177).So a floor with a glob,
$(...)or an&&chain passes on a codex job and fails on a claude job of the same manifest.floor_commandis documented only asstring(execution-manifest.md:591) and the validator checks only its type (validate-manifest.py:2174), so neither spelling is wrong - the two backends simply disagree about what a manifest string means. argv is the odd one out, and nothing chose it:shlex.splitarrived unremarked in65114b08and was carried forward in2d178edd; no commit, doc or ADR mentions argv, shell or injection.Every command now goes through the same
/bin/bash -cthe workers use.An earlier revision shelled only strings containing metacharacters. That was worse: it added a third semantics, so
pytest tests/{a,b}still meant one thing on codex and another here. The heuristic is gone.{path}substitution is nowshlex.quoted (:774), matching whatreferencing_testsalready does at:741. Paths come fromgit diffof the worker's tree and are no longer inert once every command reaches a shell.2. The receipt keeps the evidence
_run_supervisedreturns(rc, stdout_bytes); the caller wroterc, _ = …and discarded it, so a failing floor recorded a bare exit code while holding the output that explains it.The bounded tail now lands on
checks[].output_tail, with stderr captured - a broken command (sh syntax error, module-not-found,ls: cannot access) prints there and nowhere else, which is exactly the "broken command vs broken test" ambiguity this exists for.It is redacted through the canonical families imported from
compound-v-memory.py(CONVENTIONS.md: imported, never redefined), and the docstring states the honest scope: PEM /sk-/ghp_/AKIA/xox*are covered;password=, credentialed URLs and bare JWTs are not. This reduces what a committed receipt can leak; it does not eliminate it.The tail never enters
reasons[]: the emitter copies a reason matching"timeout after"intotests.failures[],previously_failing()reads that back andresolve_test_commandsre-runs it - command output there would be a path from a test's stdout to the next run's shell.3. The consumer half (emitter)
run_test_floorinitialises every result{"passed": false, "merge_blocked": true}and ADR 0003 relies on it: "already merge-blocking and already fail-closed on an empty command", Consequences: "A job reporting no test command at all is a FAIL, not a pass."agents/spec-reviewer.mdenforces the same in prose (NO_TEST_EVIDENCE- "Silence is not success").build_review_specgate 1 honours that field._job_result_fromdid not: it reads the floor only through_tests_block_from_floor, which returnsNonewhen no check carries achecker, somerge_blocked: truewas never consulted and the job kept the scope verdict'ssuccess. Same document, two consumers, opposite conclusions. Dogfood 14 moved the red half of that rule into this function; this is the empty half.tests is Noneiserror(broken machinery), notblocked.summaryinstead of falling back to the job title.successwith notestsobject.Frequency of this fail-open, stated honestly: never observed. 0 of 198 recorded results here and 0 of 17 in the reporting project. It is a latent fail-open found by reading, not a reproduced defect - an earlier revision claimed otherwise from a narrative I had not checked against the run record, and that claim is withdrawn.
Regressions considered
resolve_test_commandsputsfloor_commandfirst at every tier and raises on an empty set, so a docs-only change still resolves to[floor_command]and produces achecks[]entry;tests_blockis non-None and the empty-floor rule cannot fire.capture_stderrdefaults False, so_gitand every other caller are untouched.job_result.tests(schemaadditionalProperties: false) is unaffected:_tests_block_from_floorreads onlychecker/rc.output_taillands only inreceipts/<id>.gate.json, which V-memory does not index.Tests
fastpath --selftestPASSED; emitter 534/534 (was 526/526). Mutation-checked on both halves. Fullscripts/*.py --selftestsweep,lint-frontmatter,rules-lintgreen locally.Note on CI
The red
Full test suiteistests/test-native-points.sh, which fails 2 PRECOMPACT rows on a cleanmaintoo - the hardcoded-date time bomb that #11 fixes. Not from this branch.