fix: round eight, and the test that could not have caught it - #220
Merged
Conversation
An acceptance pass on the 0.10.0 candidate returned BLOCK with three
findings. All three are real.
The test. tests/secret-egress.test.ts called redact() on the rendered
output before asserting, so it proved redact() works on a string — never
in doubt — and could not fail for a sink that forgot to call it. Its three
"surfaces" were buildStatus, renderStatusText and buildUsage, none of which
is a sink. Every claim made for it in the CHANGELOG was false. It now
drives each sink for real and asserts on what that sink emitted; removing
redaction from any one sink fails that sink's case and no other, checked
one at a time.
Round eight. writeSse was the one HTTP egress path without redaction while
sendJson beside it had it, so the same request leaked with stream: true
and was clean with stream: false. Also unredacted: the catch-path
stderr.log write in jobs/run.ts, sibling to a success path 25 lines above
that redacts, in the very file the chokepoint edited; and the detached job
runner, which never installed the output patch and writes its own stderr
into a supervisor log nothing deletes.
The first attempt at HTTP coverage did not catch this either: the fake
upstream returns fixed text, so no secret ever reached the response path
and removing SSE redaction left the suite green. It now runs against an
endpoint that echoes the request, which is the realistic shape and the only
one that puts a credential on that path.
False positives. The registry was keyed off resolved ${VAR} references,
which covers every ${VAR} in the file — and ${VAR} is legal in any string
value. `model: ${MY_MODEL}` made that model name a redaction target and
`status` printed model=<redacted>; a harness answer mentioning it would
have been mangled silently, which is worse than the disclosure. Collection
now reads credential-bearing fields. The path-length heuristic is gone for
the same reason — it made an Azure deployment name a secret — replaced by
a config warning that names the route. Query values are taken only under a
credential-looking key. Secrets containing a JSON-escaped character are
now removed in that form too.
Verified at the built artifact: model=mock-model-v2 renders intact where it
previously read <redacted>, the api_key is still caught, and the streamed
and buffered forms of the same request are both clean.
npm run check: 1240 passed, 8 skipped, exit 0, green in one pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An acceptance pass on the 0.10.0 candidate returned BLOCK with three findings. All three were real.
The test (the one that matters)
tests/secret-egress.test.tscalledredact()on the rendered output before asserting. So it proved the redactor works on a string — never in doubt — and could not fail for a sink that forgot to call it. Its three "surfaces" werebuildStatus,renderStatusTextandbuildUsage, none of which is a sink. Every claim made for it in the CHANGELOG was false, and it is why round eight shipped green.It now drives each sink for real and asserts on what that sink emitted. Checked one sink at a time:
Round eight
writeSsewas the one HTTP egress path without redaction whilesendJsonbeside it had it — so the same request leaked withstream: trueand was clean withstream: false.Also unredacted, both found by the pass: the catch-path
stderr.logwrite injobs/run.ts, sibling to a success path 25 lines above that redacts, in the very file the chokepoint edited; and the detached job runner, which never installed the output patch and writes its own stderr into a supervisor log nothing deletes.My first attempt at HTTP coverage didn't catch this either — the fake upstream returns fixed text, so no secret ever reached the response path and removing SSE redaction left the suite green. It now runs against an endpoint that echoes the request, which is the realistic shape (an auth error naming the key it rejected) and the only one that puts a credential on that path.
False positives — the risk that matters more than the leak
The registry was keyed off resolved
${VAR}references, which covers every${VAR}in the file — and${VAR}is documented as legal in any string value. Somodel: ${MY_MODEL}made that model name a process-wide redaction target andstatusprintedmodel=<redacted>. A harness answer mentioning it would have been mangled silently: wrong work product delivered as if it were right, which is worse than the disclosure..../deployments/gpt-4-turbo-preview) a secret. A path segment can't be told from a credential by inspection, so a config warning names the route instead and the user moves it toapi_key:, where it is removable by value.?model=gemini-2.5-flashwas a target).JSON.stringifyat most sinks.Verified at the artifact
model=mock-model-v2renders intact where it previously read<redacted>; the api_key is still caught; streamed and buffered forms of the same request are both clean.npm run check: 1240 passed, 8 skipped, exit 0 — green in one pass.Still open from the pass
The changelog's two contradictory statements about
typecheck:tests, the Windows git-spawn flake inworkspace-resolve.test.ts, and the carried 0.9.0 items. Not addressed here.