W3/detector l2 classifier - #240
Merged
Merged
Conversation
The pattern layer is blind by construction to anything it has no literal for. On the tracked corpus four injection subclasses scored exactly zero on every sample at every threshold tried: paraphrase, multilingual, social and encoded. No threshold reaches a score of zero, so the fix could not be a number. llm_classifier asks a local model instead, and three of its decisions were forced by measurement rather than taste. It talks to the endpoint directly because every public entry point on LLMClient calls _guard, and a classifier living inside the guard would give inspect -> classify -> call -> _guard -> inspect with no base case. It carries its own model setting because the session provider may be hosted, and text suspected of being an attack must not leave the machine. It pins the seed because _ollama_request forwards neither temperature nor seed, and the same input was measured returning both verdicts across two consecutive calls at temperature 0. Composition is max, not sum. Measured, the layers are complementary rather than corroborating: of the five injections the model misses, three are taken by the patterns at exactly the threshold and two are missed by both. Summing would carry a sample past a threshold neither layer reached alone. l1_scorer is named rather than left as a lambda so both the evaluator and the composition compute the same number from one definition. The evaluator produces the same figures through it as before, cell for cell. Failure is open: an unreachable model, a failed status, a malformed body or an unknown verdict word all yield no opinion, never an exception. Mutation testing found the first version of the status-check test passing for the wrong reason -- an unparseable error body reached the same None through the JSON decoder -- so the body is now well-formed JSON and the transport is asserted on directly. Measured via the CLI on tests/corpus at threshold 50: L1 recall 57.1% precision 100.0% FP 0.0% (TP 28 FN 21) L1+L2 recall 95.9% precision 100.0% FP 0.0% (TP 47 FN 2) No blind subclasses remain. The two survivors are bare base64 blobs.
The one-layer report is pinned against a fresh evaluation, because the
patterns re-run anywhere. The second layer does not: CI has no ollama and a
live pass over the corpus costs minutes. Publishing 95.9% with nothing
checking it would be the shape this sprint exists to remove.
So a live run can now write down the verdicts it obtained, keyed by a hash
of the sample, and a later run can score from them instead of asking a
model. The gate replays the committed recording and compares the result to
the committed report line by line.
What that pins: the composition, the category weight, the threshold and the
pattern layer are all re-derived, so moving any of them fails the gate. What
it does not pin is the model's judgement, which is the recording itself.
Moving the prompt is still caught, because a recording carries the prompt's
fingerprint and refuses to load under a different one -- loudly, with a
non-zero exit, unlike an unreachable model. An absent model is a fact about
the machine and the layer below still holds; a recording that answers a
question the code no longer asks is a fact about the repository, and silence
there would be the defect.
A replayed run renders identically to the live run it came from: the report
names the model that produced the verdicts, not the mechanism that replayed
them. Two documents describing one measurement have to match, which is what
lets one pin the other.
An answer that could not be read is not recorded. A recording holding a
placeholder would replay a failure as a verdict, which is the one thing a
fail-open layer must never do.
Side effect worth having: anyone can now reproduce the published figure
without a GPU.
cyberai detector eval --corpus tests/corpus --l2-replay \
examples/detector-eval/l2-verdicts.json
Mutation testing on the new assertions: five predicted, five killed, and
replacing the prompt fingerprint with a constant took down six tests rather
than the two predicted. The value of a run is what falls beyond the list.
The layer measured well and reached nothing: it lived in the evaluator, and scoring a corpus is not guarding a call. CYBERAI_DETECTOR_L2 puts it inside TrustGuard.inspect, where every message bound for a provider already passes. Off unless asked for, because the price is real. Measured on this machine the layer costs about 2.4s per untrusted message, and ollama holds one model at a time: if the session model is not the classifier's, the guard also pays two model switches, about 6s each once the weights are in the page cache and half a minute cold. Defensible for one scan report, indefensible for every call in a loop. An earlier reading of that measurement was wrong -- the two tags first compared turned out to share one blob, so the run showed a model against itself and no switching at all. The model is asked only where the patterns have not already decided. Composition is max and the layer is worth one directive category, so a message at the threshold cannot move; on the eval corpus the short circuit skips 28 of 94 samples without changing a verdict. Two things surfaced here that reading the code did not produce. A mutation replacing the comparison with an unconditional assignment survived: between zero and the threshold the patterns still have an opinion, the model is asked precisely there, and a benign answer would have erased a structural match. Nothing above the threshold changes, so no corpus verdict could have exposed it. And a fail-open test raising ConnectionError rather than an httpx error failed: the except tuple did not cover socket-level failure, which would have taken down every provider call instead of one classification. A flag from this layer carries no pattern, so quarantine wraps and caps the message but substitutes no placeholder. Asserted rather than left for a reader of that branch to discover.
The security document described a detector that no longer exists alone. It now carries the two-layer figures, the command that reproduces them without a GPU, and the price of running the layer in front of a provider call. The rule that keeps figures honest was scoped to one heading, because that was the only section carrying numbers when it was written. A second layer brought a second section and a second artifact, and a rule reading one heading would have waved the new figures through unchecked -- the exact failure it exists to stop. It now reads the whole document against both committed reports. Verified by mutation: a hand-written percentage placed outside the old section fails it, and so does breaking the replay command. Tail CS is answered rather than restated. Its three patterns look for talk about base64 -- the word, "decode this", a decoder call -- and none of them look at base64. A bare blob in a header matches nothing, which is why both encoded samples score zero while the sample that spells its intent out in English is caught by other categories entirely. The weight stays with the structural group by resemblance, and stays there until a pattern that reads the encoding decides it. The hardware numbers are marked as hardware. Seconds per message and the cost of a model switch belong to the machine they were measured on; the recall figures do not.
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four injection techniques scored exactly zero on every corpus sample at every
threshold tried. Zero is not a threshold problem, so this adds a layer that is
not a list of regular expressions.
Measured at the production threshold on tests/corpus:
L1 recall 57.1% precision 100.0% FP 0.0% (TP 28 FN 21)
L1+L2 recall 95.9% precision 100.0% FP 0.0% (TP 47 FN 2)
No blind subclass remains. Both survivors are bare base64 blobs.
The layers are complementary: three of the five injections the model misses
are ones the patterns take at exactly the threshold. Composition is max, and
the model can never lower a verdict the patterns reached.
Off by default under CYBERAI_DETECTOR_L2. The published figure is reproducible
without a GPU by replaying committed verdicts, and a recording refuses to load
under a prompt it was not taken with.
What this changes
How it was measured
Checklist
ruff format --check cyberai/ tests/andruff check cyberai/ tests/passpytest -W ignore::DeprecationWarning -m "not slow and not smoke"passes