Skip to content

Add reproducible research metrics and benchmark evidence#6

Merged
CoreyLeath-code merged 7 commits into
mainfrom
agent/research-metrics-benchmarks
Jul 18, 2026
Merged

Add reproducible research metrics and benchmark evidence#6
CoreyLeath-code merged 7 commits into
mainfrom
agent/research-metrics-benchmarks

Conversation

@CoreyLeath-code

Copy link
Copy Markdown
Owner

What changed

  • replaces stale Trojan3877 badges with live badges for CoreyLeath-code/SentinelAI
  • adds a deterministic PSI/KS reference benchmark with fixed seeds and versioned raw JSON
  • records mean, median, p95, p99, min/max, throughput, peak traced memory, precision, recall, F1, and confusion matrix
  • adds a research-style benchmark report with protocol, interpretation, regression policy, provenance, and limitations
  • makes benchmark CI fail on execution/schema/quality regressions and upload raw evidence
  • updates the README metrics dashboard with evidence-backed values and explicit scope labels

Why

The existing README mixed documented performance targets with one-off smoke measurements and several badges pointed at a different repository owner. This change makes the published metrics reproducible and auditable while avoiding claims that the current evidence does not support.

Validation

  • executed benchmarks/run_benchmark.py locally with 20,000 timed evaluations and 2,000 seeded labeled samples
  • verified deterministic output schema and positive latency/throughput values
  • recorded baseline: p95 54.7 µs, p99 76.2 µs, 23,031.13 ops/s, 0.623 MiB peak traced memory
  • benchmark CI now validates F1 >= 0.95 and uploads the raw result

Important scope

The current benchmark measures the Python reference implementation of the C++ PSI/KS decision rule. It does not claim native C++, HTTP, network, database, GPU, or production accuracy performance.

@coderabbitai

coderabbitai Bot commented Jul 18, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@CoreyLeath-code, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 9 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 6661551c-59ff-4152-83ae-25610568e65a

📥 Commits

Reviewing files that changed from the base of the PR and between 1e01f8b and ec45edb.

📒 Files selected for processing (5)
  • .github/workflows/benchmarks.yml
  • README.md
  • benchmarks/benchmark_report.md
  • benchmarks/latest.json
  • benchmarks/run_benchmark.py
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch agent/research-metrics-benchmarks

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@CoreyLeath-code
CoreyLeath-code marked this pull request as ready for review July 18, 2026 20:20
@CoreyLeath-code
CoreyLeath-code merged commit 94db05d into main Jul 18, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant