Skip to content

fix(fastpath+emit): one interpreter for the test floor, and honour its verdict - #13

Merged
copeus merged 3 commits into
procoders:mainfrom
khymerao:fix/test-command-shell
Sep 11, 2026
Merged

copeus merged 3 commits into
procoders:mainfrom
khymerao:fix/test-command-shell

Conversation

@khymerao

@khymerao khymerao commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Closes defect 1 of #12 and defect 3a of #15. (#16 folded in here - it keys on the same floor document and should be bisected with this.)

Two halves of one story: what a test_contract command means to its executor, and what the floor document means to its consumer.


1. One interpreter for one command list

The plugin runs the same resolved_commands[] under two different interpreters. Every external worker runs each command through a shell, byte-identically:

compound-v-run-codex-worker.sh:189   -- /bin/bash -c "$_tc_c"
cursor:203 · antigravity:196 · opencode:237

while the claude backend's tier-1 floor ran shlex.split(raw) as argv (compound-v-fastpath-run.py:1167,1177).

So a floor with a glob, $(...) or an && chain passes on a codex job and fails on a claude job of the same manifest. floor_command is documented only as string (execution-manifest.md:591) and the validator checks only its type (validate-manifest.py:2174), so neither spelling is wrong - the two backends simply disagree about what a manifest string means. argv is the odd one out, and nothing chose it: shlex.split arrived unremarked in 65114b08 and was carried forward in 2d178edd; no commit, doc or ADR mentions argv, shell or injection.

Every command now goes through the same /bin/bash -c the workers use.

An earlier revision shelled only strings containing metacharacters. That was worse: it added a third semantics, so pytest tests/{a,b} still meant one thing on codex and another here. The heuristic is gone.

{path} substitution is now shlex.quoted (:774), matching what referencing_tests already does at :741. Paths come from git diff of the worker's tree and are no longer inert once every command reaches a shell.

2. The receipt keeps the evidence

_run_supervised returns (rc, stdout_bytes); the caller wrote rc, _ = … and discarded it, so a failing floor recorded a bare exit code while holding the output that explains it.

The bounded tail now lands on checks[].output_tail, with stderr captured - a broken command (sh syntax error, module-not-found, ls: cannot access) prints there and nowhere else, which is exactly the "broken command vs broken test" ambiguity this exists for.

It is redacted through the canonical families imported from compound-v-memory.py (CONVENTIONS.md: imported, never redefined), and the docstring states the honest scope: PEM / sk- / ghp_ / AKIA / xox* are covered; password=, credentialed URLs and bare JWTs are not. This reduces what a committed receipt can leak; it does not eliminate it.

The tail never enters reasons[]: the emitter copies a reason matching "timeout after" into tests.failures[], previously_failing() reads that back and resolve_test_commands re-runs it - command output there would be a path from a test's stdout to the next run's shell.

3. The consumer half (emitter)

run_test_floor initialises every result {"passed": false, "merge_blocked": true} and ADR 0003 relies on it: "already merge-blocking and already fail-closed on an empty command", Consequences: "A job reporting no test command at all is a FAIL, not a pass." agents/spec-reviewer.md enforces the same in prose (NO_TEST_EVIDENCE - "Silence is not success").

build_review_spec gate 1 honours that field. _job_result_from did not: it reads the floor only through _tests_block_from_floor, which returns None when no check carries a checker, so merge_blocked: true was never consulted and the job kept the scope verdict's success. Same document, two consumers, opposite conclusions. Dogfood 14 moved the red half of that rule into this function; this is the empty half.

  • The predicate is the manifest's declaration, not the resolved slice - the slice is absent exactly when resolution failed, the reachable case.
  • tests is None is error (broken machinery), not blocked.
  • The refusal carries the floor's own reasons into summary instead of falling back to the job title.
  • Unchanged: an uncontracted manifest still records success with no tests object.

Frequency of this fail-open, stated honestly: never observed. 0 of 198 recorded results here and 0 of 17 in the reporting project. It is a latent fail-open found by reading, not a reproduced defect - an earlier revision claimed otherwise from a narrative I had not checked against the run record, and that claim is withdrawn.

Regressions considered

  • No docs-only false positive. resolve_test_commands puts floor_command first at every tier and raises on an empty set, so a docs-only change still resolves to [floor_command] and produces a checks[] entry; tests_block is non-None and the empty-floor rule cannot fire.
  • capture_stderr defaults False, so _git and every other caller are untouched.
  • job_result.tests (schema additionalProperties: false) is unaffected: _tests_block_from_floor reads only checker/rc. output_tail lands only in receipts/<id>.gate.json, which V-memory does not index.
  • The selftest invariant "the recorded spelling re-parses to the argv that ran" is updated: nothing is shlex-split on the way to execution any more, so the property is now "re-runnable through the same router".

Tests

fastpath --selftest PASSED; emitter 534/534 (was 526/526). Mutation-checked on both halves. Full scripts/*.py --selftest sweep, lint-frontmatter, rules-lint green locally.

Note on CI

The red Full test suite is tests/test-native-points.sh, which fails 2 PRECOMPACT rows on a clean main too - the hardcoded-date time bomb that #11 fixes. Not from this branch.

…ord its output

Two defects in the tier-1 test floor, both found while dispatching a real run.

1. SHELL SYNTAX WAS EXECUTED AS ARGV. `shlex.split` tokenised the command and
   `_run_supervised` exec'd the token list, so every shell construct survived as a
   literal argument:

     manifest:  node --test $(ls test/*.test.js) && swiftc … && …
     argv:      ['node','--test','$(ls','test/*.test.js)','&&','swiftc', …]
     result:    rc=1

   No command substitution, no glob, no `&&` chaining. A floor that needs any of
   them fails, and the receipt makes it look like the project's tests are failing.
   A string carrying POSIX-sh metacharacters now runs through `/bin/sh -c`; a
   plain-argv string keeps the argv path unchanged, so existing manifests
   (`npm test`, `sh -c 'exit 0'`) behave exactly as before.

2. THE RECEIPT DISCARDED THE OUTPUT. `_run_supervised` returns `(rc, stdout)` and
   the caller wrote `rc, _ = …`, so a failing floor recorded only
   `tier-1: configured tests failed (rc=1): <cmd>` — nothing to act on, and
   indistinguishable from a genuine test failure. The bounded stdout tail is now
   kept on the check and in the reason. (The supervisor discards stderr; the tail
   is documented as stdout-only rather than claiming full output.)

Selftest: 8 new rows. Three are discriminators that produced the OPPOSITE verdict
before this change — `sh -c 'exit 0' && sh -c 'exit 1'` passed the floor (argv ran
only the first command) and `sh -c 'exit 1' || sh -c 'exit 0'` blocked it.
…redact

Review of the previous commit found that putting the output tail into `reasons`
opened a real path from a test's stdout to the next run's shell:

  reasons[] -> compound-v-emit-workflow.py copies any reason matching
  "timeout after" into tests.failures[] -> previously_failing() reads it back ->
  resolve_test_commands() re-runs those strings -> and a configured command now
  goes through /bin/sh.

That copy branch was effectively dead before (tier-1 reasons say "; timeout",
never "timeout after"); appending free-form command output made it reachable with
attacker-influenced content. The tail now lives ONLY on `checks[].output_tail`,
and the reason is a single line again.

Also from review:

- REDACT. The floor doc reaches receipts/<id>.gate.json, which the wave finalizer
  `git add`s, so the tail is durable and committed. It is now passed through the
  canonical PEM_RE/SECRET_RE families imported from compound-v-memory.py (the repo
  forbids a second copy of those patterns). If that import fails the text is
  dropped, not stored raw: no diagnostic is worth committing a credential.
- CAPTURE STDERR. A broken command (sh syntax error, module-not-found,
  `ls: cannot access`) prints on stderr and nothing on stdout, so the motivating
  "broken command vs broken test" case was still just `rc=N`. The supervisor
  already supports --stderr; the floor now asks for it. Off by default so `_git`
  and friends keep parseable stdout.
- HONEST DOCSTRING. The supervisor keeps the HEAD of each stream up to 256 KiB, so
  this is the tail of the first 256 KiB, not the tail of the run. Said plainly.
- `shlex.split` raises on an unbalanced quote; that now fails closed with a reason
  instead of a traceback.
- `check["via"]` records `/bin/sh -c`, which is what actually ran.

Selftest: 4 new rows (reason carries no output; redaction; stderr-only failure
produces a tail; unbalanced quote blocks). The output-tail fixture now prints a
marker that does NOT appear in the command text, so the "no output in reasons"
assertion cannot pass by accident.
…ng output

Two changes to what a `test_contract` command means and what a failing floor
records, plus the consumer half in the emitter.

1. ONE INTERPRETER. The plugin runs the SAME `resolved_commands[]` list under two
   different interpreters. Every external worker runs each command through a shell,
   byte-identically:

     compound-v-run-codex-worker.sh:189   -- /bin/bash -c "$_tc_c"
     cursor:203, antigravity:196, opencode:237

   while the claude backend's tier-1 floor ran `shlex.split(raw)` as argv. So a
   floor with a glob, `$(...)` or an `&&` chain PASSES on a codex job and FAILS on a
   claude job of the same manifest. `floor_command` is documented only as "string"
   (execution-manifest.md) and the validator checks only its type, so neither
   spelling is wrong - the two backends simply disagree. argv is the odd one out,
   and no commit or doc ever chose it: `shlex.split` arrived unremarked in 65114b0.

   Now every command goes through the same `/bin/bash -c` the workers use. An
   earlier revision shelled only strings containing metacharacters; that was worse -
   it added a THIRD semantics, so `pytest tests/{a,b}` still meant one thing on
   codex and another here.

   `{path}` substitution is now `shlex.quote`d, matching what `referencing_tests`
   already does: paths come from `git diff` of the worker's tree and are no longer
   inert once every command reaches a shell.

2. THE RECEIPT KEEPS THE EVIDENCE. `_run_supervised` returns `(rc, stdout_bytes)`
   and the caller wrote `rc, _ = ...`, so a failing floor recorded a bare exit code
   while holding the output that explains it. The bounded tail now lands on
   `checks[].output_tail`, with stderr captured too - a broken command (sh syntax
   error, module-not-found) prints there and nowhere else, which is the exact
   "broken command vs broken test" ambiguity this is for.

   It is redacted through the canonical families imported from compound-v-memory.py,
   and the docstring states the honest scope: PEM/`sk-`/`ghp_`/`AKIA`/`xox*` are
   covered, `password=`, credentialed URLs and bare JWTs are NOT. This reduces what
   a committed receipt can leak; it does not eliminate it.

   The tail never enters `reasons[]`: the emitter copies a reason matching
   "timeout after" into `tests.failures[]`, which `previously_failing()` reads and
   `resolve_test_commands` re-runs - command output there would be a path from a
   test's stdout to the next run's shell.

3. THE CONSUMER HALF (emitter). `run_test_floor` initialises every result
   `{"passed": False, "merge_blocked": True}` and ADR 0003 relies on it: "already
   merge-blocking and already fail-closed on an empty command", Consequences: "A job
   reporting no test command at all is a FAIL, not a pass." `build_review_spec`
   gate 1 honours that field; `_job_result_from` did not - it reads the floor only
   through `_tests_block_from_floor`, which returns None when no check carries a
   `checker`, so `merge_blocked: True` was never consulted and the job kept the
   scope verdict's `success`. Same document, two consumers, opposite conclusions.
   Dogfood 14 moved the RED half of that rule into this function; this is the empty
   half. The predicate is the MANIFEST's declaration, not the resolved slice, which
   is absent exactly when resolution failed. `tests is None` is `error` (broken
   machinery), not `blocked`, and the refusal carries the floor's own reasons.

   Unchanged: an uncontracted manifest still records `success` with no `tests`
   object - "no floor at all is not a failure".

Selftests: fastpath PASSED, emitter 532/532 (was 526). Mutation-checked on both
halves. Frequency of the emitter fail-open, stated honestly: never observed - 0 of
198 recorded results here and 0 of 17 in the reporting project.
@khymerao khymerao changed the title fix(fastpath): run a configured test command through a shell, and record its output fix(fastpath+emit): one interpreter for the test floor, and honour its verdict Sep 10, 2026
@copeus
copeus merged commit 157acaf into procoders:main Sep 11, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants