W3/detector corpus - #236
Merged
Merged
Conversation
The guard docstring justified DEFAULT_THRESHOLD=50 by claiming it "requires
two independent categories to agree". The arithmetic does not provide that.
risk_score is len(matches) * 25 and matches are counted per pattern, not per
category; seven of the nine categories hold more than one pattern. Measured,
"{{a}} ${b}" scores 50 on template_injection alone, and a plain HTML page
with a comment and a script tag scores 50 as well.
The false claim sat in the paragraph that justifies a production default:
prose a reviewer checks in one line, asserted by nothing. Same shape as the
version badge and the stale scorecard.
README quoted the 42-of-43 figure without the caveat the guard docstring
attaches to it and explicitly requires. That corpus is not tracked here and
no longer exists, so the figure is a recorded reason for a default, not a
reproducible measurement. README now says so.
tests/architecture/test_threshold_arithmetic.py pins the arithmetic rather
than the wording. The docstring check applies only while one category can
still reach the threshold alone; when the score is computed per unique
category the property becomes true and the check steps aside on its own
instead of having to be remembered and removed.
Mutation-tested, three mutants, all killed as predicted: scoring per unique
category, threshold raised to 100, and removal of the dollar-brace pattern.
guard.py records that its threshold and its default policy were chosen on a
corpus that is not in this repository and no longer exists on the machine
that produced it. This is that corpus, tracked, so the numbers W3 publishes
can be recomputed by anyone who clones the repo.
93 samples: 48 injections and 45 benign, one sample per file, metadata in
manifest.jsonl. Metadata is kept outside the samples on purpose. An HTML
comment is html_injection, an escape sequence is unicode_escape, a ${...}
placeholder is template_injection: three of the detector's own categories
are ordinary characters in a front-matter header, so a header inside a
sample would change what is being measured. Samples are .txt only, because
.gitignore excludes *.log and a captured nmap run saved as nmap.log would be
committed silently as nothing at all.
The injections are synthetic and each one exercises a single technique
across fifteen subclasses: direct, paraphrase without keywords, five
languages, homoglyphs, base64 and rot13, payloads split across two banners,
ANSI and bidi and zero-width smuggling, MCP tool descriptions, roleplay,
exfiltration, forged conversation turns, structured fields, code comments,
template markers, and social pressure. They were written against the attack,
not against the current regexes, so the ones this detector cannot see are
the measurement rather than a defect in the corpus.
The benign class is captured, not written: every sample is real output from
a real tool against a live target, and each entry carries the command that
produced it and the date. Ten subclasses covering nmap in four output
formats, nuclei, HTTP headers and bodies, API JSON, SQLAlchemy and Python
stack traces, container logs, and CLI tables. Three captures were discarded
rather than kept: two whatweb runs that were a ruby LoadError and one httpx
run that was a usage error. A broken tool's error message is not the output
of that tool. One capture of the environment was discarded because it
carried a username and a session path into a public repository.
test_corpus_integrity.py guards the instrument in both directions: a sample
without a manifest entry is scored by nothing, an entry without a file
changes the denominator, a captured sample without a date and an origin has
lost its provenance, either class below forty stops being a measurement, and
duplicate content would inflate both a class size and whatever metric the
copies favour, so samples are compared by content and not by name. What is
deliberately not asserted is how any sample scores. That is the measurement,
and it is what changes when the detector is rebuilt.
Mutation-tested, six mutants, all killed as predicted: a dropped manifest
line, an entry pointing at no file, a captured entry stripped of its date, a
sample overwritten with a copy of another, a sample emptied, and a class cut
below the floor.
The baseline the rebuild will be compared against, measured on the corpus committed in the previous change and recomputable by anyone who clones the repository. At the production threshold of 50: recall 25.0% over 48 injections, false positives 11.1% over 45 captured benign samples. At the detector's own is_injection cut of 25: 50.0% and 17.8%. Two findings behind those numbers matter more than the numbers. Ordinary nmap output reaches the guard. Two patterns fire on it, an XML comment and a hex escape, and two patterns are a score of 50. Nothing hostile is present in the sample. The bare phrase "disregard all previous instructions" scores zero. The pattern written for exactly that phrase is disregard (all |your |previous )? instructions, and an optional group takes one alternative: it consumes "all " and then wants "instructions" where "previous" stands. The same shape breaks print, reveal and show against "the full system prompt", "what were your original instructions", and "bypass all safety guidelines". Six patterns, one defect. The sprint plan prescribes word boundaries for this file; word boundaries do not address it. The fix is a repeating group. Three of those phrases do score 25, through the unrelated system prompt pattern that catches them by coincidence while the five patterns written for that phrasing all miss. An earlier revision of this test asserted a score of zero and went red on exactly those three, which is how the coincidence was found. Pinned separately so the rebuild treats those patterns as redundant rather than as coverage. Five subclasses score zero on every sample: paraphrase, multilingual, homoglyph, encoded and MCP tool metadata. No regex over English keywords reaches them, which is the argument for the classifier and the structural isolation layers rather than more patterns. Assertions here age in two ways on purpose. The ones pinning a defect are written to go red when it is fixed, and carry the fix instruction in the failure message; a pin that keeps passing after the thing it describes has changed is how a suite ends up documenting a codebase that no longer exists. The ones pinning a property, that the corpus is not degenerate and that injections score above benign, survive the rebuild. No individual score and neither percentage is pinned: those are the measurement. The patterns are looked up in the live INJECTION_PATTERNS by their leading word rather than copied into the test. An earlier revision copied them, and mutation testing caught it: repairing the production regex left the test green, because nothing in the assertion touched the module under test. Mutation-tested, four mutants, all killed: repeating groups on two of the broken patterns, removal of the HTML comment pattern, and the threshold lowered to 25.
…e guard's order
Two notes in the ratchet and the guard docstring described measurements
nobody could recheck. Both are now asserted against the live functions and
the tracked corpus.
The KNOWN_UNWIRED entry for scan_messages said a configurable threshold
could not be expressed through it because the score would have to be
re-derived from its details payload. That is wrong. detect_injection is
splatted into every detail, so risk_score is already sitting there.
The real obstacle is stronger than the recorded one: scan_messages scans
every message regardless of role, system included. The guard leaves system
prompts alone by design, because rewriting our own instructions is a defect
and not a defence, so wiring this helper would score the product's own
instructions as untrusted input. Its is_injection filter is also fixed at
25, so a caller at a threshold of 50 has to filter what it returns a second
time. Both halves are pinned so the note cannot go stale the way the first
version did.
The guard's docstring records that scoring the sanitised copy blinds the
detector, because sanitize_text strips {{ }} and both im_start markers and
those are three of the detector's own categories. The corpus committed
earlier in this branch now demonstrates it: fake-im-start falls from 50 to
0 and template-payload from 50 to 25 once sanitised, both across the
threshold, both invisible if the order were reversed. No benign sample
changes score at all, so scoring raw costs nothing in precision. The
docstring cited a payload scoring 75 raw and 25 through the guard; that
corpus is gone, this one is in the repository.
Mutation-tested, four mutants, all killed: scan_messages made role-aware,
the splat replaced by a single key, the im_start rule disabled in the
sanitiser, and an HTML comment rule added to it. The last one moved five
benign samples and two further injections, which is the shape of the
precision cost the guard's order avoids.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
How it was measured
Checklist
ruff format --check cyberai/ tests/andruff check cyberai/ tests/passpytest -W ignore::DeprecationWarning -m "not slow and not smoke"passes