W3/detector eval cli - #237
Merged
Merged
Conversation
…corpus The corpus committed on the previous branch is only worth what the command that reads it is worth. This is that command: it takes a corpus directory, scores every sample through the production detector, and reports precision, recall and false-positive rate per subclass. The corpus path is required and has no default, so a figure published anywhere names the corpus it came from. The report is per subclass on purpose. On the tracked corpus at the production threshold the headline is 25.0% recall against 11.1% false positives, and the headline hides the finding: seven subclasses score below the threshold on every sample they hold. Encoded payloads, homoglyphs, paraphrase without keywords, five languages, MCP tool metadata, social pressure and -- the one that stings -- exfiltration, which has five patterns written for it. Those are rendered in red and listed under the table. Rates that have no referent print as a dash rather than as zero. A subclass of captured nmap output contains no positives, so it has no precision: the first version of the table printed 0.0% there, which reads as "it fired and was always wrong" instead of "the question does not apply". Benign slices report a false-positive rate instead, which is the figure the sprint's acceptance criterion is written against. scanner_xml sits at 100% of one sample, which is the single nmap XML capture being flagged. The loader refuses a corpus it cannot trust rather than scoring a partial one: a missing manifest, an entry pointing at no file, a duplicate id, an unknown label, or a line that is not JSON each name the manifest line that caused it. A duplicate id would otherwise collapse two samples into one entry in the scores map and quietly change the denominator. evaluate() takes an optional scorer so a rebuilt detector can be measured against the same corpus without this module knowing anything about it. The default is the production path, which is what makes a run of this a statement about the product. Mutation-tested, four mutants. Three killed by the unit tests: the false-positive rate divided by the slice instead of by its negatives, the positives check counting what fired instead of what was labelled, and the flag comparison made strict, which drops true positives from 12 to 5 and shows how much of the result the boundary carries. The fourth survived and was the useful one. Unregistering the group from the root command left all 27 unit tests green while `cyberai detector eval` stopped existing: a module with a working API and no route to it is the same defect as a helper with no call site, approached from the other end. tests/unit/test_detector_eval_cli.py now drives the real Click app and all six of its tests fail under that mutant.
The detector's numbers were living in three places that a machine cannot read: a test docstring, some commit messages, and a terminal that has since scrolled away. This adds --report, which writes the Markdown the evaluation produces, and commits the file it produced. The artifact follows the convention the benchmark scorecards already set. A run-metadata table naming the timestamp, the engine version, the corpus and the threshold, then the confusion matrix, then a row per subclass, then the blind list in prose. Rates with no referent carry a dash rather than a zero, the same distinction RunMeta makes for a knob that was never measured: a placeholder in a machine-readable table reads as a value the run chose. At the production threshold on the tracked corpus: 12 true positives, 36 false negatives, 5 false positives, 40 true negatives. Recall 25.0%, precision 70.6%, false positives 11.1%. Seven injection subclasses are blind -- encoded, exfil, homoglyph, mcp_metadata, multilingual, paraphrase and social -- and the file names how many samples each one holds, so the size of what is invisible is on the page next to the headline. test_baseline_artifact_is_current.py regenerates the report in memory and compares it to the committed file line by line, timestamp excluded. Editing the file by hand fails. Changing the detector without re-running the command fails. This repository has already published a stale artifact once, a scorecard showing 28 requests where a re-run gave 21, and the rule against it was prose in a document, which gates nothing. Mutation-tested. Four against the gate, all killed: a number edited by hand in the artifact, a line of the blind list edited, undefined rates rendered as zero, and the production threshold moved, which fails three of the four because the report would then describe a product nobody ships. A fifth mutant covers the route rather than the content. The architecture gate calls render_report itself, so it would stay green if --report stopped writing anything; making the flag announce a file it never wrote fails three of the CLI tests instead.
…inst it adversarial-robustness.md said the threshold was not tuned against a corpus. That was true when it was written and stopped being true when the corpus was committed. It now carries the measurement instead, and a gate that keeps it carrying the current one. Recall 25.0%, precision 70.6%, false positives 11.1% at the production threshold; 50.0% and 17.8% at the detector's own cut of 25. The document leads with the part the headline hides: seven injection subclasses score below the threshold on every sample they hold, and no list of English regular expressions reaches any of them. It also names the two false positives that are ours, because ordinary nmap output reaches the guard on an XML comment and a hex escape, and the product flagging its own scanner is worth saying out loud in a security document. One limitation was corrected rather than removed. A single pattern hit does block a tool argument, at the sanitize_input decorator's one call site in tls_tool -- verified, not assumed -- but that path uses the detector's fixed cut of 25 while the guard uses a configurable threshold, so the two halves of the boundary answer at different sensitivities. That is the honest form of the limitation and it was not stated anywhere. The obfuscation bypass moves from an assumption to a measurement: homoglyph substitution and base64 encoding score zero on every sample in the corpus. test_docs_quote_the_artifact.py pins every percentage in the document to either the committed report or a re-run at the alternate threshold, and pins the count of blind subclasses to what the corpus actually produces. Wording is not pinned; a test that pinned sentences would fail on every paragraph edit and teach a reviewer to regenerate prose without reading it. Mutation-tested, five mutants. Four killed immediately: a figure edited in the document, the blind count edited, the reproduce command dropped, and the README row dropped. The fifth survived and was the point. Moving DEFAULT_THRESHOLD left this file green, because the blind-subclass test had 50 written into it as a literal. It imports the constant now and the mutant fails. This is the second time in two days that a test copying a production value has been caught measuring its own copy.
Recall 25.0% to 29.2% at the production threshold, precision 70.6% to 73.7%, false positives unchanged at 11.1%. At the detector's own cut of 25, recall 50.0% to 58.3%. Four injections in the corpus become visible and no benign sample moves. Three passes, in order. NFKC folds compatibility forms, which is what catches fullwidth Latin; it does not touch Cyrillic or Greek, because those are different letters and not variants of the same one, so a table of unambiguous look-alikes follows it. Zero-width characters are deleted between the two: they carry no glyph, so a payload sliced across them reads normally and matches nothing. The table is written as escape sequences, not as the characters themselves. A file holding a Cyrillic small er literally is a file a reviewer cannot read accurately, in exactly the way the attack intends, and the bidi range in this module was already written that way. test_no_confusable_letters_in_source.py keeps it so. Its scope is Cyrillic and Greek, not the whole of ASCII's complement: an earlier revision banned every non-ASCII character and went red on 180 files of em-dashes, arrows and box drawing, which would have been a demand to rewrite the repository rather than a rule anyone had. The folded text is used for matching and is never returned. The guard scores the raw message and transmits the sanitised one, and normalising on the way out would blind the detector the way scoring the sanitised copy already did once. The mapping does rewrite legitimate Cyrillic into Latin nonsense -- a Russian sentence comes out unreadable -- which costs nothing against English patterns and is the reason the copy must not escape the function. Two blind subclasses are now one fewer. homoglyph-cyrillic went from 0 to 75 and homoglyph-fullwidth from 0 to 25, so the technique is narrowed rather than closed: one of three samples still scores under the threshold. zerowidth went from 0 to 75, which takes the smuggling subclass to three of three. Six subclasses remain invisible: encoded, exfil, mcp_metadata, multilingual, paraphrase and social. The committed report and the security document were regenerated rather than edited, and the tests pinning the old figures failed until they were, which is what they are for. One of them named the sample that moved and pointed at the document to update instead of letting a subclass quietly leave the list. Mutation-tested, four mutants. Three killed: NFKC removed, the zero-width deletions removed, the confusable table emptied. The fourth survived. Returning the folded length from input_length left every test green, because the existing assertion used a pure-ASCII string where both lengths are equal by construction -- an assertion that cannot fail on the thing it claims. It now uses a string carrying zero-width characters, where the two lengths differ, and the mutant fails.
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
How it was measured
Checklist
ruff format --check cyberai/ tests/andruff check cyberai/ tests/passpytest -W ignore::DeprecationWarning -m "not slow and not smoke"passes