W3/detector l1 scoring

README.md

@@ -174,7 +174,7 @@ CyberAI is an actively developed platform, not a scaffold. Shipped and tagged:
 - **Every verdict is in the audit trail** — policy, threshold, score,
   categories and how many messages were modified, written per call. Message
   bodies stay out of it.
-- **Prompt-injection detection** — 33 patterns across 9 categories. Also run
+- **Prompt-injection detection** — 31 patterns across 10 weighted categories. Also run
   on each phase's *output*, where a hit becomes a MEDIUM finding. That pass
   is an audit signal, not a barrier: it runs after the agent has already
   called the model, and it is labelled as such in the code and the report.

cyberai/core/safety.py

@@ -3,7 +3,7 @@
 The six patterns that used to live here are gone. They were a second detector
 answering the same question as core/security/injection_detector.py, with a
 different pattern set and a different verdict, and the two disagreed: this one
-knew six patterns against that one's thirty-three. Two answers to one question
+knew six patterns against that one's thirty-one. Two answers to one question
 is not defence in depth, it is an unresolved disagreement.
 
 What is left is the half of the boundary TrustGuard does not cover. TrustGuard

cyberai/core/security/guard.py

@@ -31,15 +31,21 @@
 so 50 clears every false positive that corpus contained and costs nothing in
 recall against anything the detector could see in it.
 
-What 50 does not buy is agreement between two categories. risk_score is
-len(matches) * 25 and matches are counted per pattern, not per category:
-seven of the nine categories hold more than one pattern, so one category
-reaches 50 on its own. Measured, "{{a}} ${b}" scores 50 on template_injection
-and nothing else. An earlier revision of this docstring claimed the
-two-category property. It was false, and it was false in the paragraph that
-justifies a production default, which is the shape a reviewer checks in one
-line and nothing asserts. tests/architecture/test_threshold_arithmetic.py
-now pins the arithmetic against the claim.
+What 50 buys, since 28.08.2026, is one directive category. risk_score is
+the sum of per-category weights over the distinct categories that matched:
+50 for a category carrying an instruction to a model, 10 for one describing
+a text format. One role swap acts; two format artefacts do not, whatever
+the threshold is set to.
+
+It used to be len(matches) * 25, counted per pattern rather than per
+category, and an earlier revision of this docstring claimed that reaching
+50 meant two independent categories had agreed. That was false: seven
+categories held more than one pattern, so "{{a}} ${b}" scored 50 on
+template_injection alone. The claim sat in the paragraph justifying a
+production default, which is the shape a reviewer checks in one line and
+nothing asserts. tests/architecture/test_threshold_arithmetic.py pins the
+arithmetic now, and the property the claim described is finally true --
+by weight, not by counting.
 
 Both figures carry a caveat and are not to be quoted without it. They were
 taken while inspect() scored the *sanitised* copy, so three of the detector's

cyberai/core/security/injection_detector.py

@@ -90,7 +90,7 @@ def normalise_for_matching(text: str) -> str:
 INJECTION_PATTERNS = [
     # Role hijacking
     (r"ignore.{0,30}instructions?", "role_hijack"),
-    (r"disregard (all |your |previous )?instructions?", "role_hijack"),
+    (r"disregard (?:all |your |the |previous |prior |above )*instructions?", "role_hijack"),
     (r"forget (everything|all|your instructions)", "role_hijack"),
     (r"you are now (a |an )?(?!assistant|helpful)", "role_hijack"),
     (r"act as (a |an )?(?!assistant|helpful|security)", "role_hijack"),
@@ -100,13 +100,11 @@ def normalise_for_matching(text: str) -> str:
     (r"dan (mode|prompt)", "jailbreak"),
     (r"developer mode", "jailbreak"),
     (r"sudo (mode|prompt|access)", "jailbreak"),
-    (r"bypass (safety|filter|restriction|guideline)", "jailbreak"),
+    (r"bypass (?:all |any |the |your )*(?:safety|filter|restriction|guideline)s?", "jailbreak"),
     (r"disable (safety|filter|restriction)", "jailbreak"),
     # Data exfil via prompt
-    (r"print (your |the )?(system |full )?prompt", "exfil"),
-    (r"reveal (your |the )?(system |full )?prompt", "exfil"),
-    (r"show (me )?(your |the )?(system |full )?prompt", "exfil"),
-    (r"what (are|were) your instructions", "exfil"),
+    (r"(?:print|reveal|show)(?: me)? (?:your |the |full |entire |system )*prompt", "exfil"),
+    (r"what (?:are|were|is|was) (?:your |the |original |initial |system )*instructions?", "exfil"),
     (r"repeat (everything|all) (above|before)", "exfil"),
     # Indirect injection via external content
     (r"<\s*script", "xss_attempt"),
@@ -130,13 +128,63 @@ def normalise_for_matching(text: str) -> str:
     # Unicode / escape-sequence smuggling
     (r"\\u[0-9a-fA-F]{4}", "unicode_escape"),
     (r"\\x[0-9a-fA-F]{2}", "unicode_escape"),
-    (r"[\u202a-\u202e\u2066-\u2069]", "unicode_escape"),
+    # Bidi overrides are their own category, not a unicode_escape. The label
+    # held two opposite things: `\xNN` in a service banner is a text format
+    # and fires on four of the 45 benign samples, while an RTL override fires
+    # on none of them and on a corpus injection built to hide a payload
+    # behind it. One weight cannot serve both, and the shared label meant a
+    # bare override scored ten points.
+    (r"[\u202a-\u202e\u2066-\u2069]", "bidi_override"),
 ]
 
 COMPILED_PATTERNS = [
     (re.compile(pat, re.IGNORECASE | re.DOTALL), label) for pat, label in INJECTION_PATTERNS
 ]
 
+# What a category is worth, and the split is the whole scoring rule.
+#
+# Directive categories carry an instruction addressed to a model: a role
+# swap, a jailbreak, a request for the prompt, a forged turn boundary. One of
+# them alone reaches the production threshold of 50.
+#
+# Structural categories are artefacts of a text format. A hex escape in a
+# service banner, an XML comment in nmap's own output, a shell-style variable
+# in a Java stacktrace. Any two of them together stay at 20, below both the
+# threshold and the detector's own cut of 25.
+#
+# The split is measured, not asserted. Across the 45 benign samples captured
+# from real tools, the directive categories fire zero times and every single
+# false positive at the old scoring came from a structural one. Across the 48
+# injections the directive categories carry the signal. Weighting them apart
+# takes recall from 33.3% to 56.2% and false positives from 11.1% to 0.0% on
+# the same corpus, at the same threshold.
+#
+# encoded_payload is the one weight no sample decides: its patterns match
+# nothing in either class, so it sits with the structural group by
+# resemblance rather than by measurement. Recorded as tail CS.
+#
+# bidi_override is decided by one sample and the absence of 45. It is
+# directive because an RTL override in tool output is never a text format,
+# and because the corpus never captured one from a real tool. Splitting it
+# out changes no corpus figure -- the sample carrying it also matches other
+# categories -- so the evidence is the split itself, not a number that moved.
+# Recorded as tail CT.
+DIRECTIVE_WEIGHT = 50
+STRUCTURAL_WEIGHT = 10
+
+CATEGORY_WEIGHTS = {
+    "role_hijack": DIRECTIVE_WEIGHT,
+    "jailbreak": DIRECTIVE_WEIGHT,
+    "exfil": DIRECTIVE_WEIGHT,
+    "context_manipulation": DIRECTIVE_WEIGHT,
+    "bidi_override": DIRECTIVE_WEIGHT,
+    "html_injection": STRUCTURAL_WEIGHT,
+    "template_injection": STRUCTURAL_WEIGHT,
+    "unicode_escape": STRUCTURAL_WEIGHT,
+    "xss_attempt": STRUCTURAL_WEIGHT,
+    "encoded_payload": STRUCTURAL_WEIGHT,
+}
+
 
 def detect_injection(text: str) -> Dict[str, Any]:
     """Scan text for prompt injection patterns.
@@ -145,6 +193,18 @@ def detect_injection(text: str) -> Dict[str, Any]:
     Cyrillic look-alikes, fullwidth Latin or zero-width separators is scored
     as what it reads as. ``input_length`` stays the length of the text that
     arrived: the caller asked about that string, not about the folded one.
+
+    The score is the sum of CATEGORY_WEIGHTS over the *distinct* categories
+    that matched, capped at 100. It used to be len(matches) * 25, which
+    counted patterns: seven categories hold more than one pattern, so a
+    single technique reached the threshold by being described twice, and
+    three near-duplicate exfil patterns were worth more than one exfil
+    pattern plus a role swap. Under weights, writing another pattern for a
+    technique already covered adds no score at all.
+
+    ``matches`` is still one entry per pattern. TrustGuard's quarantine
+    policy redacts by iterating it, so collapsing it to categories would
+    take the redaction's targets away.
     """
     candidate = normalise_for_matching(text)
     matches = []
@@ -159,7 +219,8 @@ def detect_injection(text: str) -> Dict[str, Any]:
                 }
             )
 
-    risk_score = min(len(matches) * 25, 100)
+    categories = {m["type"] for m in matches}
+    risk_score = min(sum(CATEGORY_WEIGHTS.get(name, STRUCTURAL_WEIGHT) for name in categories), 100)
     is_injection = risk_score >= 25
 
     return {

docs/security/adversarial-robustness.md

@@ -54,47 +54,92 @@ the source can forge a line. Set the variable per engagement.
 
 ## Measured coverage
 
-The detector is scored against a corpus tracked in this repository, 48
+The detector is scored against a corpus tracked in this repository, 49
 injections across fifteen techniques and 45 samples of real output captured
 from real tools. Reproduce with:
 
     cyberai detector eval --corpus tests/corpus
 
-At the production threshold of 50, measured 2026-08-27 on CyberAI 1.6.0:
-recall 29.2%, precision 73.7%, false positives 11.1%. At the detector's own
-`is_injection` cut of 25: recall 58.3%, false positives 17.8%.
+At the production threshold of 50, measured 2026-08-28 on CyberAI 1.6.0:
+recall 57.1%, precision 100.0%, false positives 0.0%. The detector's own
+`is_injection` cut of 25 gives the same three figures, because no sample in
+either class scores between 25 and 50. That gap is a property of the
+weights rather than a coincidence: a directive category is worth 50 and any
+two structural ones are worth 20, so scores cluster away from the middle.
+
+A false-positive rate of zero is a statement about 45 captured samples, not
+about every tool that exists, and it should be read as the narrower claim
+it is: across that capture the categories carrying an instruction fire on
+nothing, and every false positive the old scoring produced came from a
+category describing a text format.
 
 Matching runs against a normalised copy of the text. NFKC folding, deletion
 of zero-width characters, and a table of Cyrillic and Greek letters that
 render as Latin ones. That copy is used for scoring and is never sent
 anywhere: the guard transmits the sanitised original, and normalising on the
 way out would blind the detector the way scoring the sanitised copy already
-did once. The fold costs nothing in precision on this corpus and recovers
-four injections, which is where the difference between 25.0% and 29.2%
-recall comes from.
+did once. The fold costs nothing on this corpus and carries two more
+injections across the threshold than the same patterns reach without it.
+
+Six patterns were rewritten on 2026-08-28. Each had a single optional
+qualifier group -- `disregard (all |your |previous )?instructions?` and its
+kind -- which takes one alternative and then demands its object, so
+"disregard all previous instructions" matched nothing. A repeating group
+matches them. Recall moved by four points and no benign sample changed
+score, which is the honest size of the win: the phrasings a published
+bypass list would have used were already covered by other patterns.
+
+The larger move on the same day was to the score itself. It was
+`len(matches) * 25`, which counted patterns rather than techniques: a
+category described in two patterns reached the threshold on its own, and
+three near-duplicate exfil patterns outweighed a genuine role swap. The
+score is now the sum of per-category weights over the distinct categories
+that matched. Categories carrying an instruction to a model -- role
+hijacking, jailbreak, prompt exfiltration, forged turn boundaries -- are
+worth 50 each; categories describing a text format -- XML comments,
+template markers, hex escapes, script tags -- are worth 10, so any two of
+them together stay below both cuts.
+
+The cost is recorded with the gain. One corpus injection is built from a
+template marker and nothing else; it scores 10 now and is no longer
+detected. That is the trade the measurement argues for: one crafted sample
+against every stacktrace, HTML body and nmap comment in the benign half.
 
 The overall recall figure is still the least useful number in that
-paragraph. Six injection subclasses score below the threshold on every
-sample they hold: encoded payloads, exfiltration phrasing, MCP tool
-metadata, five non-English languages, paraphrase that avoids the keywords,
-and social pressure. A list of English regular expressions cannot reach any
-of them, which is the case for a layer that is not a list of regular
-expressions rather than for more entries in this one.
-
-Two false positives are worth naming because they are ours. Ordinary
-`nmap -sV` output scores 50 and reaches the guard, on an XML comment and a
-hex escape, with nothing hostile present; the XML output format does the
-same. The product flags its own scanner.
+paragraph. Four injection subclasses score below the threshold on every
+sample they hold: encoded payloads, five non-English languages, paraphrase
+that avoids the keywords, and social pressure. A list of English regular
+expressions cannot reach any of them, which is the case for a layer that is
+not a list of regular expressions rather than for more entries in this one.
+
+It was six. Exfiltration phrasing and MCP tool metadata left the list when
+the weights changed, and neither left because a pattern was added: their
+samples already matched one directive category and scored 25, which the
+threshold of 50 discarded. Two whole techniques were invisible for the
+arithmetic's sake rather than for want of a rule.
+
+Until 2026-08-28 the false positives worth naming were ours. Ordinary
+`nmap -sV` output scored 50 and reached the guard, on an XML comment and a
+hex escape, with nothing hostile present; the XML output format did the
+same. The product flagged its own scanner. Both categories are structural
+and the same samples now score 20: still seen, no longer acted on. That is
+the intended shape -- the detector keeps reporting what it matched, and the
+score decides what any of it is worth.
 
 ## Known Limitations
 
 - Pattern-based injection detection is bypassable with obfuscation. This is
   measured, not assumed: base64 encoding scores zero on every sample in the
-  corpus. Homoglyph substitution is now folded before matching, and one of
-  three samples reaches the threshold rather than none, so the fold narrows
-  the bypass without closing it.
+  corpus. Homoglyph substitution is folded before matching and all three of
+  those samples now clear the threshold, which closes that bypass on the
+  corpus without closing it in general -- the fold maps the confusables it
+  knows about.
+- A structural signal alone is never a verdict, by construction. A payload
+  assembled entirely from template markers, HTML comments or hex escapes
+  scores at most 20 whatever it says. This is the deliberate half of the
+  weighting and the half an attacker can aim at.
 - One detector answers for the whole project. `core/safety.py` used to carry a
-  second one, six patterns against the canonical thirty-three; it now reports
+  second one, six patterns against the canonical thirty-one; it now reports
   the canonical verdict and holds no patterns of its own.
 - A single pattern hit blocks a tool argument, at the `sanitize_input`
   decorator's one call site. That path uses the detector's own cut of 25 and
@@ -115,7 +160,7 @@ same. The product flags its own scanner.
 ## Future Work
 
 - Enforce KB namespace boundaries, or state plainly that the KB is shared.
-- Semantic injection detection (LLM-based classifier). The seven blind
+- Semantic injection detection (LLM-based classifier). The four blind
   subclasses above are the argument for it and the corpus is the instrument
   that will say whether it helped.
 - Read-only agent mode for passive recon.

examples/detector-eval/baseline.md

@@ -1,30 +1,30 @@
 # Detector Evaluation
 
-**recall 29.2% — false positives 11.1%**
+**recall 57.1% — false positives 0.0%**
 
 ## Run metadata
 
 | field | value |
 | --- | --- |
-| timestamp | 2026-08-27T18:59:43Z |
+| timestamp | 2026-08-28T08:15:00Z |
 | engine version | CyberAI 1.6.0 |
 | corpus | tests/corpus |
 | threshold | 50 |
-| injections | 48 |
+| injections | 49 |
 | benign | 45 |
 
 ## Overall
 
 | metric | value |
 | --- | --- |
-| true positives | 14 |
-| false negatives | 34 |
-| false positives | 5 |
-| true negatives | 40 |
-| precision | 73.7% |
-| recall | 29.2% |
-| f1 | 41.8% |
-| false positive rate | 11.1% |
+| true positives | 28 |
+| false negatives | 21 |
+| false positives | 0 |
+| true negatives | 45 |
+| precision | 100.0% |
+| recall | 57.1% |
+| f1 | 72.7% |
+| false positive rate | 0.0% |
 
 ## Per-subclass breakdown
 
@@ -34,37 +34,35 @@ A slice holding no positives has no precision, and one holding no negatives has
 | --- | --- | --- | --- | --- | --- |
 | api_json | 11 | 0 | -- | -- | 0.0% |
 | cli_table | 7 | 0 | -- | -- | 0.0% |
-| code_context | 2 | 1 | 100.0% | 50.0% | -- |
+| code_context | 2 | 2 | 100.0% | 100.0% | -- |
 | config_json | 1 | 0 | -- | -- | 0.0% |
 | container_logs | 3 | 0 | -- | -- | 0.0% |
-| context_forgery | 3 | 2 | 100.0% | 66.7% | -- |
-| direct | 4 | 2 | 100.0% | 50.0% | -- |
+| context_forgery | 3 | 3 | 100.0% | 100.0% | -- |
+| direct | 4 | 4 | 100.0% | 100.0% | -- |
 | encoded | 3 | 0 | -- | 0.0% | -- |
-| exfil | 4 | 0 | -- | 0.0% | -- |
-| homoglyph | 3 | 1 | 100.0% | 33.3% | -- |
-| html_body | 3 | 2 | -- | -- | 66.7% |
+| exfil | 4 | 3 | 100.0% | 75.0% | -- |
+| homoglyph | 3 | 3 | 100.0% | 100.0% | -- |
+| html_body | 3 | 0 | -- | -- | 0.0% |
 | http_headers | 6 | 0 | -- | -- | 0.0% |
-| mcp_metadata | 4 | 0 | -- | 0.0% | -- |
+| mcp_metadata | 4 | 2 | 100.0% | 50.0% | -- |
 | multilingual | 5 | 0 | -- | 0.0% | -- |
 | paraphrase | 5 | 0 | -- | 0.0% | -- |
-| roleplay | 3 | 1 | 100.0% | 33.3% | -- |
-| scanner_text | 8 | 2 | -- | -- | 25.0% |
-| scanner_xml | 1 | 1 | -- | -- | 100.0% |
+| roleplay | 3 | 3 | 100.0% | 100.0% | -- |
+| scanner_text | 8 | 0 | -- | -- | 0.0% |
+| scanner_xml | 1 | 0 | -- | -- | 0.0% |
 | service_json | 2 | 0 | -- | -- | 0.0% |
-| smuggling | 3 | 3 | 100.0% | 100.0% | -- |
+| smuggling | 4 | 4 | 100.0% | 100.0% | -- |
 | social | 3 | 0 | -- | 0.0% | -- |
 | split | 2 | 1 | 100.0% | 50.0% | -- |
 | stacktrace | 3 | 0 | -- | -- | 0.0% |
-| structured | 2 | 1 | 100.0% | 50.0% | -- |
-| template | 2 | 2 | 100.0% | 100.0% | -- |
+| structured | 2 | 2 | 100.0% | 100.0% | -- |
+| template | 2 | 1 | 100.0% | 50.0% | -- |
 
 ## Blind subclasses
 
 Every sample in these scored below the threshold. This is what an overall recall figure cannot show, and it is the argument for a layer that is not a list of regular expressions.
 
 - `encoded` — 0 of 3 flagged
-- `exfil` — 0 of 4 flagged
-- `mcp_metadata` — 0 of 4 flagged
 - `multilingual` — 0 of 5 flagged
 - `paraphrase` — 0 of 5 flagged
 - `social` — 0 of 3 flagged