ci: benchmark regression gate on every PR - #92
Merged
Conversation
Three checks run after the test suite passes: - Representative fixture: fn must be 0 (recall = 1.0) - Combined representative+clean: precision >= 0.90, recall = 1.0 - Actual dataset: fp must be 0 (no new false positives) Uses the JSON output mode added in v0.7.0; python3 is available on ubuntu-latest without any extra setup step. https://claude.ai/code/session_014T1x8ZiDbJcVvkZBfP91nk
- README.md: update detection pass count (4→5), AC needle count (161→188), signal count (21→28); add 5 missing signals to table (protocol_violation, unexpected_tool_sequence, runtime_credential_access, unexpected_network_call, annotation_deception); update credential_reference to mention API key prefixes; update injection payload category count (8→10) to include SSRF and ReDoS; add --from-config examples to Quick Start and Usage sections; add config.rs to architecture diagram; fix fuzzer module counts (28 signals, 188 patterns, 37 response patterns, 10 payload categories); add config-file audit to roadmap as shipped - bench/README.md: add v0.13 column to version table; update AC pattern count (161→188) in section header and Pass 1 description; add API key prefixes to credential_reference signal row - CHANGELOG.md: add benchmark regression CI gate entry (#92) - CONTRIBUTING.md: require ./bench/run.sh before every PR; document the three CI gate checks enforced by the benchmark job https://claude.ai/code/session_014T1x8ZiDbJcVvkZBfP91nk
Automates the documentation gap-finding and update process: - Phase 0 extracts ground-truth counts from source (signals, AC needles, response patterns, payload categories, corpus records) - Phase 1 audits README.md, bench/README.md, CONTRIBUTING.md, CHANGELOG.md against those counts in parallel - Phase 2 applies fixes file-by-file with explicit checklists for each file Invoke with /docs-update after any release or significant feature merge. https://claude.ai/code/session_014T1x8ZiDbJcVvkZBfP91nk
/add-signal — guides adding a new detection signal end-to-end: 7 required sites in src/fuzzer/mod.rs (enum, ALL, as_str, rule_id, description, tags, Display), AC needle placement in description.rs or response.rs, benchmark before/after, corpus record, CHANGELOG entry, and a final checklist that mirrors the CI gates /release-prep — guides cutting a release: version validation, CHANGELOG promotion, Cargo.toml bump, bench/README.md version column, README version references, full local CI suite, commit/tag/push, and release.yml verification checklist https://claude.ai/code/session_014T1x8ZiDbJcVvkZBfP91nk
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
benchmarkCI job that runs after tests pass and blocks the PR if detection quality regressesbench/mcptox_representative.json, 44 tools) —fnmust be 0 (full recall on the canonical attack set)bench/clean_tools.json, 64 tools) — precision ≥ 0.90, recall = 1.0 (catches new false positives on clean tools)bench/mcptox_actual.json, 485 tools) —fpmust be 0 (no false positives on the real-world dataset)--from-configusage examples, v0.13 benchmark column, missing signals in signal table, API key prefixes incredential_reference).claude/commands/:/docs-update— three-phase repeatable process: extract ground-truth counts from source, audit docs in parallel, apply fixes file-by-file/add-signal— end-to-end checklist for adding a new signal: all 7 required sites insrc/fuzzer/mod.rs, AC needle placement, benchmark before/after, corpus record, CHANGELOG entry/release-prep— release checklist: version validation, CHANGELOG promotion,Cargo.tomlbump,bench/README.mdversion column, full local CI suite, commit/tag/push, release.yml verificationCurrent numbers (as of main after merging #89–#91)
Test plan
benchmarkjob passes (all three gate checks green)fmt,clippy,test,buildjobs all passfuzzd benchmark --schema bench/mcptox_representative.json --output jsonreturnsfn: 0fuzzd benchmark --schema bench/mcptox_actual.json --output jsonreturnsfp: 0/docs-update,/add-signal,/release-prepskills listed and invocable in Claude Codehttps://claude.ai/code/session_014T1x8ZiDbJcVvkZBfP91nk