[Submission] Add tau-bench leaderboard adapter - #192
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a new tau_bench adapter under utils/ that ingests the public tau-bench leaderboard submissions (remote or local replay) and emits Every Eval Ever EvaluationLog records with per-domain pass@k and cost metrics, plus associated metadata. Includes unit tests and documents the adapter in utils/README.md.
Changes:
- Introduces
utils/tau_bench/adapter.pyto fetch/parse tau-bench leaderboard manifests + submissions and convert them into EEE logs/results. - Adds
tests/test_tau_bench_adapter.pycovering schema validation, metric mapping, voice metadata preservation, local manifest loading, and error handling. - Registers the new adapter in
utils/README.mdand addsutils/tau_bench/__init__.py.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
| utils/tau_bench/adapter.py | New adapter implementation converting tau-bench submissions to EEE logs/results. |
| utils/tau_bench/init.py | Adds package marker/docstring for the new adapter module. |
| utils/README.md | Documents the new tau_bench adapter in the adapters table. |
| tests/test_tau_bench_adapter.py | Adds tests for adapter correctness and schema compatibility. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| evaluation_timestamp = evaluation_date(submission) | ||
| version = ( | ||
| (submission.get('methodology') or {}).get('tau2_bench_version') | ||
| ) or 'unknown' |
There was a problem hiding this comment.
The adapter consumes source JSON without validating it against tau-bench’s submission schema first. Rather than adding separate isinstance(..., dict) guards everywhere, could we validate each submission.json once at the loader boundary and then let the conversion code assume the documented shape?
That would also catch malformed model_release, voice_config, domain results, required fields, and manifest values consistently. We should probably keep the explicit finite/range checks for scores too, since Python’s JSON handling and custom inputs can still introduce edge cases.
| methodology = submission.get('methodology') or {} | ||
| value = methodology.get('evaluation_date') or submission.get( | ||
| 'submission_date' | ||
| ) |
| methodology = submission.get('methodology') or {} | ||
| voice_config = submission.get('voice_config') or {} | ||
| pipeline = ( |
| def run(args: argparse.Namespace) -> int: | ||
| records = load_submissions( | ||
| input_dir=args.input_dir, | ||
| base_url=args.base_url, | ||
| sections=args.sections, | ||
| ) | ||
| if args.limit is not None: | ||
| records = records[: args.limit] | ||
| bundles = make_logs(records) | ||
| paths = export_logs(bundles, args.output_dir) | ||
| for path in paths: | ||
| print(path) | ||
| return len(paths) |
|
it looks like this is fine to be approved, but there are a couple of notes that can be implemented either now or later; they are not of utmost importance and do not block this feature right now:
|
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
mrshu
left a comment
mrshu
left a comment
There was a problem hiding this comment.
⚒️ review-anvil report
Review decision: COMMENT — The adapter is clear and well tested, and four data-contract details would benefit from follow-up.
Result: One high-priority metric identity issue and three medium-priority provenance or validation issues remain.
Scope: This PR adds a tau-bench leaderboard adapter for remote and local submissions.
Checks: 4 concerns checked; 3 confirmed, 1 narrowed, 0 ruled out, 0 set aside.
Second check: targeted, 2 reviewers; 4 kept, 3 fix paths clarified, 0 set aside, 0 removed.
Earlier review comments
Earlier review comments (5 items)
- Fixed: Schema-version test now uses
adapter.SCHEMA_VERSION. - Still present at an outdated anchor:
tau2_bench_versionstill assumes truthymethodologyvalues are mappings. - Still present at an outdated anchor:
evaluation_datekeeps the same malformed-input assumption. - Still present at an outdated anchor: Generation config keeps the same malformed-input assumption.
- Still present at an outdated anchor: Remote
--limitstill slices after all selected submissions are downloaded.
What I noticed
- [high] metric semantics
utils/tau_bench/adapter.py:389— Tau-bench Pass^k values use a pass@k identifier. Upstream computes all-k success aspass_hat_k, while pass@k means at least one success. Downstream metric joins can combine incompatible scores. (RAVF001; inline) - [medium] provenance
utils/tau_bench/adapter.py:197— Remote records cite mutablemain, and local replay invents the same upstream URL for local bytes. The exported record cannot identify the exact input that produced its scores. (RAVF002; inline) - [medium] score validation
utils/tau_bench/adapter.py:596— Local and custom input accepts booleans, non-finite values, out-of-range Pass^k percentages, and negative costs. Current official submissions are valid, but supported replay inputs can emit invalid scores. (RAVF003; inline) - [medium] developer identity
utils/tau_bench/adapter.py:91— The current upstreamZ.aisubmission falls back to developer slugz.ai. Other repository mappings usezhipu-ai, so the same provider splits across identities and output paths. (RAVF004; inline)
Things to try
Things to try (4 items)
- [high] metric semantics — The global
pass_hat_kidentifier can match the existing repository mapping while keeping tau-bench names, result IDs, andkparameters. (RAVF001) - [medium] provenance — The default GitHub source can resolve one commit. Custom sources and local replay can use content hashes without claiming unrelated upstream URLs. (
RAVF002) - [medium] score validation — Pass metrics can enforce finite 0–100 values, while cost can enforce finite non-negative values. Boolean inputs need a separate rejection. (
RAVF003) - [medium] developer identity — Adding the exact
Z.aialias to the existing map would keep both bundle developer and model ID consistent. (RAVF004)
Run details
- Target: PR #192 (
benshi34/tau-bench-adapter, 4 files, +897/-0) - Rounds: 1/1 completed; adaptive off; material findings remained
- Mix: 3 codex-exec
- Focus: correctness, upstream format behavior, provenance, maintainability, and constructive suggestion-oriented language
- Earlier review comments: 5 comments; 4 still present and 1 fixed
- Finding counts: 0 critical, 1 high, 3 medium, 0 low, 0 nit
- Checks: concerns=4; confirmed=3/narrowed=1/ruled-out=0/set-aside=0
- Second check: targeted; reviewers=2; kept=4/clarified=3/set-aside=0/removed=0; approval changed no
- Set aside: 0 items
Reviewed with review-anvil.
| f'tau-bench {domain} Pass^{k} success rate reported on ' | ||
| 'the public leaderboard.' | ||
| ), | ||
| metric_id='tau_bench.pass_at_k', |
There was a problem hiding this comment.
[high] metric semantics — Pass^k is identified as pass@k
Upstream computes this value as pass_hat_k: all selected trials must succeed. Pass@k instead means at least one succeeds. Using the pass@k ID can merge mathematically different scores in downstream joins.
The repository already uses the global pass_hat_k ID for Pass^k. Keeping the existing display name and k parameter would preserve tau-bench context.
| submission=submission, | ||
| source_url=submission_source_url(submission_id), | ||
| ) | ||
| ) |
There was a problem hiding this comment.
[medium] provenance — Local replay is attributed to mutable upstream main
Remote mode does not record a resolved revision. Local mode also builds the current upstream URL from only the submission ID, even when local bytes differ. The record therefore cannot identify the exact source that produced its scores.
The default GitHub source can resolve one commit per run. Custom and local inputs can keep their supplied provenance plus a content hash without claiming unrelated upstream bytes.
| if raw is None or raw == '': | ||
| return None | ||
| try: | ||
| return float(raw) |
There was a problem hiding this comment.
[medium] score validation — Supported replay input can emit values outside the source contract
float() accepts booleans, NaN, infinity, Pass^k values outside 0–100, and negative costs. Current official submissions are valid, but local replay and custom sources can reach this path.
Pass metrics can use finite inclusive 0–100 checks. Cost can use a finite non-negative check, with booleans rejected before numeric conversion.
| 'Qwen': 'qwen', | ||
| 'Sierra': 'sierra', | ||
| 'xAI': 'xai', | ||
| 'Zhipu AI': 'zhipu-ai', |
There was a problem hiding this comment.
[medium] developer identity — The current Z.ai submission gets a second developer slug
The organization map handles Zhipu AI but not Z.ai, so fallback slugging creates z.ai. Other repository adapters and developer helpers use zhipu-ai for the same provider.
Adding the exact Z.ai alias to this existing map would keep the bundle developer, model ID, and output path aligned.
Summary
Adds a tau-bench adapter that fetches the public leaderboard submissions manifest and per-submission JSON from
sierra-research/tau2-bench, then converts populated domain metrics into Every Eval Ever aggregate records.The adapter supports:
pass_1throughpass_4--input-dirreplay for review/testingNo instance-level trajectory export is included in this first version.
Validation
uv run ruff check utils/tau_bench/adapter.py tests/test_tau_bench_adapter.pyuv run pytest tests/test_tau_bench_adapter.pyuv run python -m utils.tau_bench.adapter --input-dir /Users/ben.sierra.ai/Documents/tau2-bench/web/leaderboard/public/submissions --output-dir /private/tmp/eee-tau-bench-smoke --limit 3uv run python -m every_eval_ever validate /private/tmp/eee-tau-bench-smokeuv run python -m utils.tau_bench.adapter --output-dir /private/tmp/eee-tau-bench-live-smoke --limit 1uv run python -m every_eval_ever validate /private/tmp/eee-tau-bench-live-smoke