W2/honest provenance

CHANGELOG.md

@@ -17,6 +17,30 @@ All notable changes to CyberAI are documented here.
   The `LICENSE` file holds the canonical Apache text unmodified; the copyright
   line moved to `NOTICE`.
 
+### Fixed
+
+- **Benchmark contamination.** The exploitation engine held a literal from a
+  target this project wrote, and the SQL-injection proof accepted the success
+  body our own bench login prints — so part of a published 4/4 measured
+  recognition rather than exploitation, and shipped to PyPI inside the engine.
+  Both proofs are now structural: a traversal reads a shape the target should
+  not serve, an auth bypass is proven by the 401 to 200 transition. An
+  architecture test runs every shipped proof against every string the bench
+  apps are built from, which is what caught the case a grep could not see.
+  Both scorecards were regenerated against live targets: still 4/4, at 21
+  requests instead of 28. Full account in
+  `docs/benchmarks/contamination-2026-08.md`.
+- **Scorecard provenance.** `provider` and `model` defaulted to the string
+  `unspecified` and the CLI passed neither, so every published card named a
+  value the run had chosen. The rows are omitted when nothing was measured.
+  The version row was keyed `engine`, which the CLI also writes to name the
+  engine that ran, giving one card two rows under one key; it is now
+  `engine version`, and a second writer reaching an occupied key raises.
+- **Cause of a zero call count.** A run whose code path constructs no LLM
+  client was reported as `no_api_key_for_openai`, pointing the reader at a
+  credential that would change nothing. The path now answers for itself, above
+  the credential causes.
+
 ### Added
 
 - **Contributor License Agreement** (`CLA.md`), adapted from the Apache ICLA

cyberai/bench/agent_engine.py

@@ -42,6 +42,8 @@
 from cyberai.bench.runner import BenchResult, BenchTask
 from cyberai.bench.targets import LocalSuiteAdapter, VulnTarget
 from cyberai.core.config import CyberAIConfig
+from cyberai.core.cost_tracker import CostTracker
+from cyberai.core.llm_usage import llm_zero_reason
 from cyberai.core.scan_session import ScanSession
 
 logger = logging.getLogger(__name__)
@@ -56,6 +58,12 @@ class AttackOutcome:
     requests_sent: int = 0
     findings: list[dict[str, Any]] = field(default_factory=list)
     oob_confirmed: int = 0
+    # What the model did, or None for not measured. Zero and absent are
+    # different facts: a run that proves no model was reached can say so,
+    # while a run with no way to count must not publish a zero it never
+    # counted. Both stay None unless the attacker establishes otherwise.
+    llm_calls: Optional[int] = None
+    llm_zero_reason: Optional[str] = None
 
     @property
     def solved(self) -> bool:
@@ -130,15 +138,37 @@ def agent_attack(
     description = str((task.metadata if task else {}).get("one_day_description", ""))
     classes = classes_from_description(description) if one_day and description else None
 
-    ReconAgent(cfg, session)._run_web_recon(base_url)
-    report = ExploitAgent(cfg, session)._run_web_exploit(base_url, classes=classes)
+    recon = ReconAgent(cfg, session)
+    recon._run_web_recon(base_url)
+    exploit = ExploitAgent(cfg, session)
+    report = exploit._run_web_exploit(base_url, classes=classes)
+
+    # Read off the agents that ran, not off the config. Both are constructed
+    # with two positional arguments, so the client parameter keeps its None
+    # default and no model can be reached from here -- a fact of this code
+    # path, which is why the count is a proven zero rather than a default.
+    # The moment either agent is handed a client, there is no tracker on
+    # this path to count its calls with, so both fields go back to None:
+    # not measured is the only answer available until one exists.
+    model_free = recon.llm is None and exploit.llm is None
 
     return AttackOutcome(
         confirmed=int(report.get("confirmed", 0)),
         endpoints_tested=int(report.get("endpoints_tested", 0)),
         requests_sent=int(report.get("requests_sent", 0)),
         findings=list(report.get("findings", [])),
         oob_confirmed=int(report.get("params_oob_confirmed", 0)),
+        llm_calls=0 if model_free else None,
+        llm_zero_reason=(
+            llm_zero_reason(
+                cfg.llm,
+                CostTracker(),
+                client_built=False,
+                engine_uses_a_model=False,
+            )
+            if model_free
+            else None
+        ),
     )
 
 
@@ -210,6 +240,8 @@ def _run(task: BenchTask) -> BenchResult:
                 "endpoints_tested": outcome.endpoints_tested,
                 "requests_sent": outcome.requests_sent,
                 "findings": outcome.findings,
+                "llm_calls": outcome.llm_calls,
+                "llm_zero_reason": outcome.llm_zero_reason,
                 "judge_solved": judged,
                 "agreement": None if judged is None else outcome.solved == judged,
             }

cyberai/bench/cve_bench_runner.py

@@ -109,6 +109,8 @@ def _run(task: BenchTask) -> BenchResult:
                 "endpoints_tested": outcome.endpoints_tested,
                 "requests_sent": outcome.requests_sent,
                 "findings": outcome.findings,
+                "llm_calls": outcome.llm_calls,
+                "llm_zero_reason": outcome.llm_zero_reason,
             }
             if status is None:
                 # No verdict means no measurement. Say so instead of scoring it.

cyberai/bench/run_manifest.py

@@ -7,7 +7,10 @@
   - a content hash over the suite's tasks (id/name/criteria) — proves the suite
     wasn't quietly swapped to an easier one between runs,
   - the run config (model, provider, temperature, seed) — the knobs that affect
-    outcome,
+    outcome. A knob that was never measured is recorded as null, not as a
+    placeholder string: "unspecified" reads as a value the run chose, and a
+    probe engine that never contacts a model would publish it as if a model
+    had been involved,
   - a manifest hash over all of the above — a single fingerprint to compare runs.
 
 `set_global_seed` pins Python's `random` (and PYTHONHASHSEED for child procs) so
@@ -53,13 +56,21 @@ def hash_tasks(tasks: list[BenchTask]) -> str:
 
 @dataclass(frozen=True)
 class RunConfig:
-    """The knobs that affect a run's outcome."""
-
-    model: str = "unspecified"
-    provider: str = "unspecified"
-    temperature: float = 0.0
+    """The knobs that affect a run's outcome.
+
+    Everything a caller may leave unmeasured defaults to None, so the manifest
+    distinguishes "this run did not involve a model" from "this run used a
+    model named unspecified". temperature is not exempt: 0.0 is a real setting
+    a caller can choose, and a default of 0.0 would claim deterministic
+    sampling for a run that never sampled anything. seed keeps a concrete
+    default because set_global_seed always pins one.
+    """
+
+    model: str | None = None
+    provider: str | None = None
+    temperature: float | None = None
     seed: int = DEFAULT_SEED
-    max_iterations: int = 0
+    max_iterations: int | None = None
     extra: dict[str, Any] = field(default_factory=dict)
 
 

cyberai/bench/scorecard.py

@@ -28,11 +28,21 @@
 @dataclass(frozen=True)
 class RunMeta:
     """Provenance for a scorecard run. All fields optional/defaulted so a
-    scorecard can be produced even in minimal/CI contexts."""
+    scorecard can be produced even in minimal/CI contexts.
+
+    A knob that was never measured is None and its row is left out, rather
+    than published as "unspecified": a placeholder in a machine-readable
+    table reads as a value the run chose, and an engine that never contacts
+    a model would name one as if it had. llm_calls is the same distinction
+    on the other side -- zero says a model was proven not to have been
+    reached, absent says nothing counted it.
+    """
 
     engine_version: str = __version__
-    model: str = "unspecified"
-    provider: str = "unspecified"
+    model: str | None = None
+    provider: str | None = None
+    llm_calls: int | None = None
+    llm_zero_reason: str | None = None
     note: str = ""
     extra: dict[str, Any] = field(default_factory=dict)
 
@@ -126,7 +136,12 @@ def _run_metric_lines(report: SuiteReport) -> list[str]:
 
 
 def generate_scorecard(report: SuiteReport, meta: RunMeta | None = None) -> str:
-    """Render a Markdown scorecard for one suite run."""
+    """Render a Markdown scorecard for one suite run.
+
+    The version row is keyed `engine version`. It used to be `engine`, which
+    the CLI also writes to name the engine that ran, so one published card
+    carried two rows under one key: `CyberAI 1.5.0` and `agent`.
+    """
     meta = meta or RunMeta()
     lines: list[str] = []
     lines.append(f"# Benchmark Scorecard — `{report.suite}`")
@@ -138,13 +153,29 @@ def generate_scorecard(report: SuiteReport, meta: RunMeta | None = None) -> str:
     lines.append("| field | value |")
     lines.append("| --- | --- |")
     lines.append(f"| timestamp | {_utc_now_iso()} |")
-    lines.append(f"| engine | CyberAI {meta.engine_version} |")
-    lines.append(f"| provider | {meta.provider} |")
-    lines.append(f"| model | {meta.model} |")
+    lines.append(f"| engine version | CyberAI {meta.engine_version} |")
+    rows: list[tuple[str, str]] = []
+    if meta.provider:
+        rows.append(("provider", meta.provider))
+    if meta.model:
+        rows.append(("model", meta.model))
+    if meta.llm_calls is not None:
+        rows.append(("llm calls", str(meta.llm_calls)))
+    if meta.llm_zero_reason:
+        rows.append(("llm zero reason", meta.llm_zero_reason))
     if meta.note:
-        lines.append(f"| note | {meta.note} |")
-    for k, v in meta.extra.items():
-        lines.append(f"| {k} | {v} |")
+        rows.append(("note", meta.note))
+    rows += [(str(k), str(v)) for k, v in meta.extra.items()]
+    written = {"timestamp", "engine version"}
+    for key, value in rows:
+        if key in written:
+            raise ValueError(
+                f"duplicate scorecard metadata key: {key!r}. The table is read "
+                "by machines, so one key carrying two meanings is a defect, not "
+                "a formatting choice."
+            )
+        written.add(key)
+        lines.append(f"| {key} | {value} |")
     lines.append("")
     lines.append("## Per-class breakdown")
     lines.append("")

cyberai/cli/bench.py

@@ -155,6 +155,28 @@ def _second_opinion(details: dict) -> bool | None:
     return None
 
 
+def _model_participation(report) -> tuple[int | None, str | None]:
+    """What the whole run can say about the model, or nothing.
+
+    Per-task facts only roll up when they agree. A run where some tasks
+    reached a model and others could not has no single answer, and picking
+    either one publishes a number the run did not produce -- so the split
+    itself is what travels. Absent everywhere stays absent: a scorecard with
+    no row is honest about not having measured, a row reading `unknown` is
+    a value.
+    """
+    results = list(report.results)
+    if not results:
+        return None, None
+    proven = [r for r in results if r.details.get("llm_calls") == 0]
+    if not proven:
+        return None, None
+    if len(proven) == len(results):
+        reasons = {str(r.details.get("llm_zero_reason")) for r in proven}
+        return 0, reasons.pop() if len(reasons) == 1 else "mixed_reasons"
+    return None, f"mixed: {len(proven)} of {len(results)} tasks reached no model"
+
+
 def _select_tasks(tasks: list, wanted: tuple[str, ...]) -> list:
     """Narrow a suite to the requested ids, or fail loudly.
 
@@ -326,7 +348,16 @@ def run(
             extra["filtered"] = f"{len(selected)} of {len(all_tasks)} tasks: " + ", ".join(
                 t.id for t in selected
             )
-        md = generate_scorecard(report, RunMeta(note="cyberai bench run", extra=extra))
+        calls, reason = _model_participation(report)
+        md = generate_scorecard(
+            report,
+            RunMeta(
+                note="cyberai bench run",
+                extra=extra,
+                llm_calls=calls,
+                llm_zero_reason=reason,
+            ),
+        )
         out = Path(scorecard_path)
         out.parent.mkdir(parents=True, exist_ok=True)
         out.write_text(md)

cyberai/core/llm_usage.py

@@ -25,6 +25,7 @@ def llm_zero_reason(
     *,
     client_built: bool,
     dry_run: bool = False,
+    engine_uses_a_model: bool = True,
 ) -> Optional[str]:
     """Why no LLM call happened, or None when at least one did.
 
@@ -36,13 +37,23 @@ def llm_zero_reason(
     same zero as a run that never asked. `attempts` separates them: a non-zero
     attempt count with no call is a refusal, and it outranks the other causes
     because it is the one thing measured directly.
+
+    `engine_uses_a_model` is answered by the code path, not by the config, and
+    it is checked before the credential causes. A path that constructs its
+    agents without a client cannot be fixed by a key: the default provider is
+    a cloud one and the default key is absent, so the missing-key cause fires
+    on every such run and points the reader at a knob that changes nothing.
+    Measured causes still outrank it -- a recorded answer means a model spoke,
+    whatever the path claims about itself.
     """
     if tracker.call_count:
         return None
     if tracker.attempts:
         return "provider_refused"
     if dry_run:
         return "dry_run"
+    if not engine_uses_a_model:
+        return "engine_uses_no_model"
     provider = llm_config.provider
     if provider in ("openai", "anthropic") and not llm_config.api_key:
         return f"no_api_key_for_{provider}"

docs/benchmarks/contamination-2026-08.md

@@ -0,0 +1,87 @@
+# Benchmark contamination, August 2026
+
+The local suite scored 4/4 on both engines for months. Part of that score was
+self-referential: the exploitation engine held a literal from a target this
+project wrote, so at least one proof measured recognition rather than
+exploitation. This page records what was wrong, what changed, and what the
+numbers did.
+
+## What was contaminated
+
+`cyberai/agents/exploit/web_payloads.py` carried a module constant holding the
+exact flag `cyberai/bench/apps/path_traversal.py` plants outside its web root,
+and the SQL-injection proof accepted the JSON body our own bench login prints
+on success. Both shipped to PyPI inside the exploitation engine, so every user
+received an engine that recognised strings from this repository's CTF apps.
+
+The second form is the instructive one. It carried no flag at all: the proof
+looked for a status field that our bench login returns and that Juice Shop
+also returns from an untouched product listing. A grep for `FLAG{` finds the
+first and never the second.
+
+## What changed
+
+| Before | After |
+|---|---|
+| Traversal proof matched a planted flag literal | Structural match on the shape of a file the target should not serve |
+| SQLi proof accepted our login's success body | Auth bypass proven by the 401 to 200 transition, not by any string |
+| Nothing prevented a recurrence | `tests/architecture/test_no_bench_leak.py` runs every production proof against every string constant the bench apps are built from |
+
+The guard asserts two things. The textual half forbids a flag literal outside
+`cyberai/bench/`. The behavioural half is what caught the harder case: it
+requires that no shipped proof is satisfied by any literal our targets are
+built from. A grep cannot express the second.
+
+## What the numbers did
+
+Both engines still score 4/4. The honest expectation before the run was a
+drop -- one traversal payload was removed and the SQLi proof became stricter --
+and the drop did not happen. The targets remain solvable by proofs that know
+nothing about them.
+
+The cost of solving them fell:
+
+| Task | Requests, published 2026-08-17 | Requests, 2026-08-26 | In-band proofs |
+|---|---|---|---|
+| local-sqli-login | 12 | 5 | 2, unchanged |
+| local-cmdi-ping | 3 | 3 | 1, unchanged |
+| local-path-traversal | 3 | 3 | 1, unchanged |
+| local-ssrf-fetch | 10 | 10 | 0 in band, 1 out of band |
+| **total** | **28** | **21** | **4 in band, 1 out of band** |
+
+The whole difference is one task. A proof that no longer accepts a string this
+project plants settles the parameter earlier than one that did, so the walk
+stops sooner and spends less.
+
+That saving belongs to the decontamination work, not to the run that published
+these cards: the same task measured on `main` before the provenance changes
+also spends 5. The published card predated the decontamination merge and had
+never been regenerated -- which is its own finding, and the reason the cards
+are now regenerated in the same branch that changes anything they report.
+
+## What the score does not say
+
+The agent engine reaches no model. `ReconAgent` and `ExploitAgent` are
+constructed with two positional arguments on this path, so the client
+parameter keeps its `None` default and no call can be made. The card records
+this directly: `llm calls: 0`, `llm zero reason: engine_uses_no_model`.
+
+So 4/4 is the score of a deterministic exploit corpus cross-checked by an
+independent probe. It is not a measurement of model-driven exploitation, and
+reading it as one would overstate what runs here. The metadata block exists so
+a reader does not have to take that on trust.
+
+The probe engine's card carries no call row at all. It builds no agents and
+counts nothing, and a zero there would claim a measurement nobody took.
+
+## Reproducing
+
+```bash
+cyberai bench run --suite local --engine agent \
+  --scorecard examples/local-bench/scorecard-agent.md
+```
+
+Measured 2026-08-26, seed 1337, zero-day mode, all four targets up, the blind
+target confirmed through a live out-of-band collector. See
+[reproducibility.md](reproducibility.md) for what a run pins and what it still
+does not measure.