Skip to content

perf: add real HTTP client benchmarks - #75

Merged
medz merged 4 commits into
mainfrom
perf/add-benchmarks
Jun 23, 2026
Merged

perf: add real HTTP client benchmarks#75
medz merged 4 commits into
mainfrom
perf/add-benchmarks

Conversation

@medz

@medz medz commented Jun 23, 2026

Copy link
Copy Markdown
Owner

Summary

Add a local benchmark harness for Oxy and comparable Dart HTTP clients:

  • bench/main.dart is the single runner entrypoint with quick/full modes, suite selection, filtering, and JSON output.
  • bench/baseline.dart measures lower-level JSON, bytes, ht.Headers, and ht.Body costs.
  • bench/oxy.dart, bench/http.dart, and bench/dio.dart compare real client requests against the same loopback HttpServer.
  • bench/src/server.dart serves fixed GET/POST scenarios and drains plus validates POST request bodies, so upload paths are included in the comparison.
  • README positioning no longer promotes alternative HTTP libraries from Oxy's own feature list.

Client suites now use the same real network scenarios: GET /empty, GET /json, GET /bytes-64k, POST /json, and POST /bytes-64k. This keeps the comparison simple: same local server, same payloads, no mock transports, no custom Dio adapter, and no middleware/interceptor tuning.

This remains a local/manual benchmark tool rather than a CI performance gate.

No linked issue; this is a standalone performance benchmark PR.

Validation

  • dart format --output=none --set-exit-if-changed bench/main.dart bench/oxy.dart bench/http.dart bench/dio.dart bench/src/server.dart
  • dart analyze
  • dart bench/main.dart --quick --json
  • dart bench/main.dart --quick --suite=clients --json
  • dart bench/main.dart --suite=clients --json
  • dart test -p vm
  • dart pub publish --dry-run

Summary by CodeRabbit

  • New Features

    • Added comprehensive HTTP benchmarking suite supporting multiple HTTP clients with test cases for GET/POST requests, JSON payload handling, and binary data transfers
    • Includes configurable benchmark runner with command-line options and reporting formats
  • Documentation

    • Minor README clarifications
  • Chores

    • Added benchmark-related development dependencies

@coderabbitai

coderabbitai Bot commented Jun 23, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@medz, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 35 minutes and 15 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits.

🚦 How do rate limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan refill rate.

For paid Pro and Pro+ PR reviews, CodeRabbit uses rolling per-developer review limits. Reviews become available again as older review attempts age out of the rolling limit window.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 29839d6e-65c5-48b7-91f4-984bd39b7878

📥 Commits

Reviewing files that changed from the base of the PR and between 319d2e2 and 9b8aaf7.

📒 Files selected for processing (4)
  • bench/dio.dart
  • bench/http.dart
  • bench/oxy.dart
  • bench/src/network.dart
📝 Walkthrough

Walkthrough

Adds a full benchmark infrastructure to the bench/ directory: a runner framework (BenchmarkCase, SyncBenchmark, AsyncBenchmark, RunConfig, measureSuites), shared data fixtures, a local HttpServer routing five endpoints, four suites (baseline, oxy, http, dio), and a CLI entrypoint. Three dev dependencies are added. Four words are removed from the README.

Changes

Benchmark Infrastructure

Layer / File(s) Summary
Benchmark framework and shared data
bench/src/runner.dart, bench/src/data.dart
Defines BenchmarkCase interface, SyncBenchmark/AsyncBenchmark wrappers, BenchmarkResult, RunConfig (CLI parsing, suite/filter matching, warmup/measure timing), measureSuites, printReport/jsonReport, and consume black-hole helper. data.dart supplies deterministic 1 KB/64 KB byte arrays, a JSON payload map/text, and precomputed header pair lists.
Local benchmark HTTP server
bench/src/server.dart
Implements startBenchmarkServer/closeBenchmarkServer lifecycle, binds an ephemeral loopback HttpServer, and routes GET /empty, GET /json, GET /bytes-64k, POST /json, POST /bytes-64k with body-drain validation; returns 400 on size mismatch and 404 for unknown routes.
Baseline (non-network) suite
bench/baseline.dart
Adds baselineSuite with sync benchmarks for JSON encode/decode, Uint8List copy, Headers construction at 8/32 entries, header iteration, Body construction, Body clone, and an async Body.bytes() read.
Network suites: oxy, http, dio
bench/oxy.dart, bench/http.dart, bench/dio.dart
Adds oxySuite (Oxy Client), httpSuite (package:http), and dioSuite (dio), each with five async benchmark cases covering GET /empty, GET /json, GET /bytes-64k, POST /json, and POST /bytes-64k. Each suite uses a lazily initialized singleton client and exposes a close function.
CLI entrypoint and dev dependencies
bench/main.dart, pubspec.yaml
bench/main.dart registers all four suites, parses CLI args, conditionally starts/closes the benchmark server and per-suite clients, errors when no cases match, and prints the report. pubspec.yaml adds benchmark_harness, dio, and http as dev dependencies.

README Cleanup

Layer / File(s) Summary
Remove transitional sentence
README.md
Deletes the "another package may be a better fit" sentence from the "Why Choose Oxy" section.

Sequence Diagram(s)

sequenceDiagram
  participant CLI as bench/main.dart
  participant Server as BenchmarkServer
  participant Suite as oxySuite / httpSuite / dioSuite
  participant Runner as measureSuites

  CLI->>Server: startBenchmarkServer()
  CLI->>Runner: measureSuites(config, suites)
  Runner->>Suite: measureCase(RunConfig)
  Suite->>Server: HTTP GET/POST /empty, /json, /bytes-64k
  Server-->>Suite: 200 response (body/status)
  Suite-->>Runner: BenchmarkResult(runtimeMicros)
  Runner-->>CLI: List<BenchmarkResult>
  CLI->>Suite: closeOxyBenchmarks() / closeHttpBenchmarks() / closeDioBenchmarks()
  CLI->>Server: closeBenchmarkServer()
  CLI->>CLI: printReport(config, results)
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Suggested labels

feature, dependencies

Poem

🐇 Hop hop, the benchmarks bloom,
JSON flies and bytes zoom!
A server spins on loopback's loop,
Dio, http, oxy — all in the group.
The rabbit clocks each microsecond tight,
And README shines a little more bright. ✨

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and specifically summarizes the main addition: real HTTP client benchmarks for the Oxy library.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch perf/add-benchmarks

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@medz medz changed the title perf: add benchmark baselines perf: add HTTP client benchmark suites Jun 23, 2026
@medz
medz marked this pull request as ready for review June 23, 2026 13:24

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 650c09f8b2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread bench/oxy.dart Outdated
@medz medz changed the title perf: add HTTP client benchmark suites perf: add real HTTP client benchmarks Jun 23, 2026
@coderabbitai coderabbitai Bot added dependencies Dependency updates feature New capability or public behavior labels Jun 23, 2026
@medz

medz commented Jun 23, 2026

Copy link
Copy Markdown
Owner Author

@codex review

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@bench/oxy.dart`:
- Around line 11-49: The GET request benchmarks in oxy.dart (get-empty,
get-json-decode, and get-bytes-64k) are missing the headers that are included in
the POST requests and in the corresponding benchmarks in http.dart and dio.dart,
making cross-client comparisons unfair. Add the same headers parameter to all
GET requests that are already being used in the post-bytes-64k request (such as
_octetHeaders) to ensure consistent request shapes across all client test
suites.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 9c5ecd27-d504-415e-8d5f-95ff9c43ab63

📥 Commits

Reviewing files that changed from the base of the PR and between 72ed7b4 and 319d2e2.

📒 Files selected for processing (10)
  • README.md
  • bench/baseline.dart
  • bench/dio.dart
  • bench/http.dart
  • bench/main.dart
  • bench/oxy.dart
  • bench/src/data.dart
  • bench/src/runner.dart
  • bench/src/server.dart
  • pubspec.yaml
💤 Files with no reviewable changes (1)
  • README.md

Comment thread bench/oxy.dart

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 319d2e2511

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread bench/dio.dart Outdated
Comment thread bench/http.dart
Comment thread bench/oxy.dart Outdated
Comment thread bench/dio.dart Outdated
@medz
medz merged commit 5d7509f into main Jun 23, 2026
3 checks passed
@medz
medz deleted the perf/add-benchmarks branch June 23, 2026 14:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Dependency updates enhancement feature New capability or public behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant