W3/detector l1 scoring - #239
Merged
Merged
Conversation
Six patterns used a single optional qualifier group. An optional group takes one alternative and then demands its object, so `disregard (all |your |previous )?instructions?` matched "disregard instructions" and "disregard all instructions" but not "disregard all previous instructions" -- the phrasing every published bypass list uses. The same shape broke print/reveal/show ... prompt on "the full system prompt", "what (are|were) your instructions" on any inserted adjective, and "bypass (safety|filter|...)" on "bypass all safety guidelines". A repeating group takes as many qualifiers as are present. The fix is the quantifier, not a word boundary: W3.3 of the sprint plan prescribes \b, which does not address this at all. Measured on the tracked corpus at the production threshold of 50: recall 29.2% -> 33.3% (true positives 14 -> 16) precision 73.7% -> 76.2% false positives 11.1%, unchanged -- no benign sample changed score The honest size of the win is four points, not the defect's apparent severity: three of the seven phrasings already scored above zero through the unrelated `system prompt` pattern, which caught them by coincidence while every pattern written for that phrasing missed. The blind subclasses are the same six. Exfil stays blind because its samples now match one exfil pattern and score 25, and the threshold is 50: that is arithmetic, not phrasing, and is the next commit's problem. tests/architecture/test_detector_baseline.py pinned the defect and went red on all seven phrases, as it was written to. It is inverted here into a regression guard against the old shape returning, and its prefix lookup now keys on the leading word rather than on `what (are|were)`, which was pattern syntax and broke as soon as that group gained an alternative. examples/detector-eval/baseline.md regenerated by the command, never by hand; docs/security/adversarial-robustness.md and the baseline pin in tests/unit/test_eval_corpus.py updated to the measured figures. Closes tail CL.
`print ... prompt`, `reveal ... prompt` and `show ... prompt` were three patterns differing only in the verb. Now that each absorbs its qualifiers, one alternation covers all three, and the detector holds 31 patterns instead of 33. The claim that they were redundant is measured, not argued. The evaluation report regenerated after the removal is identical to the one before it, cell for cell, timestamp aside: none of the 93 corpus samples changed score. Three patterns were doing no work a single one does not do. The removal matters beyond tidiness because risk_score is len(matches) * 25. Near-duplicate patterns inflate the score of any text that trips them all, so redundancy is not free -- it is a silent weight on whichever phrasing happens to have the most patterns written for it. Rescoring is the next commit; removing the duplicates first keeps that measurement from inheriting this one's distortion. README, docs/security/adversarial-robustness.md and the module docstring of cyberai/core/safety.py name the pattern count in prose and are pinned against it by tests/architecture/test_documented_pattern_count.py, which went red on all three and is the reason they are updated here rather than discovered stale in a month. CHANGELOG's "33 patterns" is left alone: it describes the release that shipped 33. Closes tail CM.
… count risk_score was len(matches) * 25, which counted patterns rather than techniques. Seven categories held more than one pattern, so a single technique reached the production threshold of 50 by being described twice, while a technique described once could not reach it at all. The threshold's own docstring had claimed for months that 50 meant two categories agreed; it never did. The score is now the sum of per-category weights over the distinct categories that matched, capped at 100. Categories that carry an instruction addressed to a model -- role hijacking, jailbreak, prompt exfiltration, forged turn boundaries, bidi overrides -- are worth 50 each. Categories that describe a text format -- XML comments, template markers, hex escapes, script tags -- are worth 10, so any two together stay below every cut. The split is measured, not reasoned. Across the 45 benign samples captured from real tools the directive categories fire zero times, and every false positive the old scoring produced came from a structural one: recall 33.3% -> 56.2% (true positives 16 -> 27) precision 76.2% -> 100.0% false positives 11.1% -> 0.0% (5 -> 0) blind subclasses six -> four Two subclasses left the blind list without a pattern being added. Exfil phrasing and MCP tool metadata already matched one directive category and scored 25; the threshold of 50 discarded them. Two techniques were invisible for the arithmetic's sake. Ordinary nmap output, the false positive this repository has been carrying in writing since day 11, now scores 20 instead of 50. Still matched, still reported, no longer acted on. Recorded costs, because they are real: - One corpus injection is built from a template marker and nothing else. It scores 10 and is no longer detected. tests/architecture/test_security_wired pins that so it cannot be read as a win. - encoded_payload's weight is decided by no sample in either class. It sits with the structural group by resemblance. Tail CS. - bidi_override was split out of unicode_escape, which held two opposite things under one label: `\xNN` in a service banner fires on four benign samples, an RTL override fires on none and on an injection built to hide behind it. The split moves no corpus figure -- the evidence is the two meanings, not a number. Tail CT. Eleven tests went red and eight were predicted. The three that were not each found something: an orchestrator test proved a bare RTL override scored ten points, and the CLI threshold test proved its own second assertion had become undemonstrable -- the blind subclasses score exactly zero, not merely below the cut, so no threshold reaches them. That is a sharper claim than the one it replaced and it is the argument for L2. Closes tails CN and CP.
Mutation testing on the weights introduced in the previous commit found a hole. Removing bidi_override from CATEGORY_WEIGHTS -- dropping an RTL override from 50 points to the structural default of 10 -- was noticed by exactly two tests, both in test_orchestrator.py, neither written for it. The two weights that were covered failed 41 and 9 tests respectively. The corpus could not notice either. Its only bidi sample carries "print your system prompt" in plain text and scores 100 through exfil and context_manipulation, so the override contributes nothing that the other categories do not already provide. A category introduced and a sample that cannot contradict it: producer without consumer, applied to data. Two consumers added, one on each side of the boundary: - tests/corpus/injections/bidi-only.txt, an RTL override in an SSH banner with no other signal in the text. Scores 50 on bidi_override alone. - test_a_bare_bidi_override_reaches_the_threshold_alone, which asserts the same property directly rather than through a corpus figure. The corpus is 49 injections now and the published numbers move with it: recall 56.2% -> 57.1% over the larger denominator, precision and false positives unchanged at 100.0% and 0.0%. The report is regenerated by the command and docs/security/adversarial-robustness.md follows it. The finding is worth more than the fix. A weight is a decision, and a decision no test can contradict is one nobody recorded -- reasoning about what an RTL override means is not the same as measuring that the code treats it that way. The remaining weight in that position is encoded_payload, whose patterns match nothing in either class, so its value is still resemblance rather than measurement. Tail CS stays open.
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four commits, each measured on the tracked corpus before and after.
(all |your |previous )?,which takes one alternative and then demands its object, so
"disregard all previous instructions" matched nothing. Recall 29.2% -> 33.3%.
identical cell for cell, which is the proof they were redundant. 33 -> 31.
len(matches) * 25countedpatterns, not techniques. Directive categories (role hijack, jailbreak,
exfil, forged turn boundaries, bidi overrides) are worth 50; structural ones
(XML comments, template markers, hex escapes, script tags) are worth 10.
Across 45 benign samples captured from real tools the directive categories
fire zero times. Recall 33.3% -> 56.2%, false positives 11.1% -> 0.0%.
removing it was noticed by two tests neither written for it, and by nothing
in the corpus. Added a sample where the override is the only signal.
Closes tails CL, CM, CN, CP. Opens CS (encoded_payload's weight is decided by
no sample) and CT (bidi_override's split moved no corpus figure).
Recorded costs: one corpus injection built from a template marker alone is no
longer detected, and the false-positive rate of zero is a statement about 45
captured samples, not about every tool that exists.
What this changes
How it was measured
Checklist
ruff format --check cyberai/ tests/andruff check cyberai/ tests/passpytest -W ignore::DeprecationWarning -m "not slow and not smoke"passes