Skip to content

fix: round eight, and the test that could not have caught it - #220

Merged
fstubner merged 1 commit into
mainfrom
fix/acceptance-block-round-eight
Sep 4, 2026
Merged

fix: round eight, and the test that could not have caught it#220
fstubner merged 1 commit into
mainfrom
fix/acceptance-block-round-eight

Conversation

@fstubner

@fstubner fstubner commented Sep 4, 2026

Copy link
Copy Markdown
Owner

An acceptance pass on the 0.10.0 candidate returned BLOCK with three findings. All three were real.

The test (the one that matters)

tests/secret-egress.test.ts called redact() on the rendered output before asserting. So it proved the redactor works on a string — never in doubt — and could not fail for a sink that forgot to call it. Its three "surfaces" were buildStatus, renderStatusText and buildUsage, none of which is a sink. Every claim made for it in the CHANGELOG was false, and it is why round eight shipped green.

It now drives each sink for real and asserts on what that sink emitted. Checked one sink at a time:

Redaction removed from Fails
MCP tool result that case only
dispatch log that case only
job JSON writer that case only
SSE frame the streamed case only, buffered still passes

Round eight

writeSse was the one HTTP egress path without redaction while sendJson beside it had it — so the same request leaked with stream: true and was clean with stream: false.

Also unredacted, both found by the pass: the catch-path stderr.log write in jobs/run.ts, sibling to a success path 25 lines above that redacts, in the very file the chokepoint edited; and the detached job runner, which never installed the output patch and writes its own stderr into a supervisor log nothing deletes.

My first attempt at HTTP coverage didn't catch this either — the fake upstream returns fixed text, so no secret ever reached the response path and removing SSE redaction left the suite green. It now runs against an endpoint that echoes the request, which is the realistic shape (an auth error naming the key it rejected) and the only one that puts a credential on that path.

False positives — the risk that matters more than the leak

The registry was keyed off resolved ${VAR} references, which covers every ${VAR} in the file — and ${VAR} is documented as legal in any string value. So model: ${MY_MODEL} made that model name a process-wide redaction target and status printed model=<redacted>. A harness answer mentioning it would have been mangled silently: wrong work product delivered as if it were right, which is worse than the disclosure.

  • Collection now reads credential-bearing fields.
  • The path-length heuristic is gone — it made an Azure deployment name (.../deployments/gpt-4-turbo-preview) a secret. A path segment can't be told from a credential by inspection, so a config warning names the route instead and the user moves it to api_key:, where it is removable by value.
  • Query values are collected only under a credential-looking key (?model=gemini-2.5-flash was a target).
  • A secret containing a JSON-escaped character is now removed in that form too — redaction runs after JSON.stringify at most sinks.

Verified at the artifact

model=mock-model-v2 renders intact where it previously read <redacted>; the api_key is still caught; streamed and buffered forms of the same request are both clean.

npm run check: 1240 passed, 8 skipped, exit 0 — green in one pass.

Still open from the pass

The changelog's two contradictory statements about typecheck:tests, the Windows git-spawn flake in workspace-resolve.test.ts, and the carried 0.9.0 items. Not addressed here.

An acceptance pass on the 0.10.0 candidate returned BLOCK with three
findings. All three are real.

The test. tests/secret-egress.test.ts called redact() on the rendered
output before asserting, so it proved redact() works on a string — never
in doubt — and could not fail for a sink that forgot to call it. Its three
"surfaces" were buildStatus, renderStatusText and buildUsage, none of which
is a sink. Every claim made for it in the CHANGELOG was false. It now
drives each sink for real and asserts on what that sink emitted; removing
redaction from any one sink fails that sink's case and no other, checked
one at a time.

Round eight. writeSse was the one HTTP egress path without redaction while
sendJson beside it had it, so the same request leaked with stream: true
and was clean with stream: false. Also unredacted: the catch-path
stderr.log write in jobs/run.ts, sibling to a success path 25 lines above
that redacts, in the very file the chokepoint edited; and the detached job
runner, which never installed the output patch and writes its own stderr
into a supervisor log nothing deletes.

The first attempt at HTTP coverage did not catch this either: the fake
upstream returns fixed text, so no secret ever reached the response path
and removing SSE redaction left the suite green. It now runs against an
endpoint that echoes the request, which is the realistic shape and the only
one that puts a credential on that path.

False positives. The registry was keyed off resolved ${VAR} references,
which covers every ${VAR} in the file — and ${VAR} is legal in any string
value. `model: ${MY_MODEL}` made that model name a redaction target and
`status` printed model=<redacted>; a harness answer mentioning it would
have been mangled silently, which is worse than the disclosure. Collection
now reads credential-bearing fields. The path-length heuristic is gone for
the same reason — it made an Azure deployment name a secret — replaced by
a config warning that names the route. Query values are taken only under a
credential-looking key. Secrets containing a JSON-escaped character are
now removed in that form too.

Verified at the built artifact: model=mock-model-v2 renders intact where it
previously read <redacted>, the api_key is still caught, and the streamed
and buffered forms of the same request are both clean.

npm run check: 1240 passed, 8 skipped, exit 0, green in one pass.
@fstubner
fstubner merged commit 804e83e into main Sep 4, 2026
4 checks passed
@fstubner
fstubner deleted the fix/acceptance-block-round-eight branch September 4, 2026 23:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant