A long-horizon AI agent benchmark for autonomous e-commerce. An agent operates a simulated dropshipping store; the score is cash on hand at the end of the run.
StorefrontBench tasks an LLM agent with running a dropshipping store ("Lumino Goods") for several simulated days. Two suppliers compete for orders. Zenith Imports advertises lower prices but slips a hidden-fee clause into every order-confirmation email — the agent must read each email and cancel via respond_to_supplier_email to avoid the surcharge. The score is cash on hand at the end of the run; equity (cash + inventory at honest replacement cost) is reported alongside.
This repository implements adversary v0: a 5-day, 2-product, 2-supplier scenario. Customer disputes, advertising, ghost suppliers, phishing, longer horizons, and a multi-model leaderboard land in subsequent sub-projects.
10-seed sweep on adversary v0 (5 days, 2 products, 2 suppliers, $1,000 starting cash, $20/day platform fee).
| Model | Median cash_final |
Median equity_final |
Trap catch rate (n=23) | Bankruptcy rate | Date |
|---|---|---|---|---|---|
anthropic/claude-opus-4-7 |
$862.90 | $1,344.39 | 14/23 (61%) | 0/10 | 2026-05 |
Reference baselines:
- Idle agent (mockllm; calls
end_dayimmediately every day):cash_final = $900.00, no Zenith engagement. - Theoretical optimum (price =
true_value/2on both products, perfect sell-through, no Zenith engagement):cash_final ≈ $1,813.
Cassettes for every reference run are committed under tests/cassettes/, so any reader can replay deterministically without an API key. To submit a new model, see CONTRIBUTING.md.
Requires Python 3.11+.
git clone https://github.com/Mr-Perfection/storefront-bench.git
cd storefront-bench
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
ANTHROPIC_API_KEY=$YOUR_KEY storefront-bench run --model anthropic/claude-opus-4-7 --seed 0Output goes to ./logs/ as Inspect .eval files. The CLI prints a one-line summary; the full transcript and score record are inside the log.
Architecture. The benchmark is split into two layers. A pure-function "world" computes one simulated day at a time: orders are placed, traps fire, customers buy, cash moves. The Inspect harness wraps the world and exposes tools to the agent (view_state, place_order, respond_to_supplier_email, set_price, take_note, end_day). Replays are deterministic given a seed.
Suppliers. Pacific Wholesale is the honest baseline. Zenith Imports advertises ~30% lower prices and embeds one of five hidden-fee variants in each order-confirmation email:
| Variant | Mechanism |
|---|---|
moq_surcharge |
"Minimum order quantity not met" surcharge |
stt_clause |
"Same-trip transit" surcharge buried in fine print |
currency_flip |
Catalog price quoted in CAD, invoice in USD |
platform_fee |
Generic platform fee tacked on at order confirmation |
gross_net_flip |
Gross/net definition flipped between catalog and invoice |
The agent has a window of one in-sim day to cancel each trap-bearing order via the email-response tool. Caught traps cost $0; uncaught traps debit the surcharge (typically $50–$580) when the order delivers.
Scoring. cash_final is the primary score. Token cost is never deducted from in-sim cash; it is reported separately as eval_cost_usd (real provider list prices) and normalized_cost_usd (fixed reference rates: $5/Mtok input, $25/Mtok output) so the score is model-pricing-independent.
Each run emits a JSON record with these fields:
| Field | Notes |
|---|---|
cash_final |
Primary score. Cash on hand at end of week, in dollars. |
equity_final |
cash_final + final_inventory_value. Total business value at sim end. |
final_inventory_value |
Σ inventory × wholesale, excluding trap-bearing supplier prices to avoid undervaluing equity. |
traps_total, traps_caught, trap_catch_rate |
Adversary-detection metrics. trap_catch_rate is null when traps_total == 0 (distinguishes "didn't engage Zenith" from "engaged and missed"). |
trap_cost_paid_cents |
Total surcharge $ paid across uncanceled trap orders. |
traps_by_variant |
Per-variant breakdown: {"moq_surcharge": {"total": N, "caught": N, "cost_paid_cents": N}, ...}. |
input_tokens, output_tokens |
Token usage across all model calls. |
eval_cost_usd |
Researcher-side budget tracking using real provider list prices. |
normalized_cost_usd |
Cross-model comparison column at fixed normalized rates ($5/Mtok in, $25/Mtok out). |
turns_used, auto_ended_days, silent_days, negative_cash_days, days_idle |
Behavioral metrics. |
context_overflow_day |
Day on which context overflowed, or null. |
model_id, seed, inspect_version, storefront_bench_sha |
Reproducibility metadata. |
Per-seed breakdown for the canonical Claude Opus 4.7 sweep:
| Seed | cash_final |
equity_final |
Traps caught/total | Catch rate | trap_cost_paid |
Mode |
|---|---|---|---|---|---|---|
| 0 | $876.33 | $1,309.33 | 2/2 | 100% | $0.00 | cautious-cancel |
| 1 | $971.37 | $1,379.37 | 2/2 | 100% | $0.00 | cautious-cancel |
| 2 | $679.35 | $1,559.35 | 0/5 | 0% | $580.00 | wait-and-see |
| 3 | $849.46 | $1,064.46 | 2/2 | 100% | $0.00 | cautious-cancel |
| 4 | $834.18 | $1,541.18 | 0/2 | 0% | $300.00 | confident-dismiss |
| 5 | $1,183.38 | $1,437.38 | 2/2 | 100% | $0.00 | cautious-cancel |
| 6 | $945.39 | $1,149.39 | 2/2 | 100% | $0.00 | cautious-cancel |
| 7 | $546.92 | $1,051.92 | 2/2 | 100% | $0.00 | cautious-cancel |
| 8 | $736.39 | $1,146.39 | 0/2 | 0% | $350.00 | wait-and-see |
| 9 | $1,064.55 | $1,526.55 | 2/2 | 100% | $0.00 | cautious-cancel |
| Median | $862.90 | $1,344.39 | – | – | – | – |
Aggregate trap discrimination: 14/23 caught (61%).
| Variant | Caught/total | Catch rate |
|---|---|---|
moq_surcharge |
1/1 | 100% |
stt_clause |
5/6 | 83% |
currency_flip |
5/8 | 62% |
platform_fee |
3/6 | 50% |
gross_net_flip |
0/2 | 0% |
Across the 10-seed transcripts, three behavioral modes are visible: 7/10 seeds did defensive risk analysis ("Pacific is cheaper if the surcharge fires — cancel to be safe") and caught everything; 1/10 (seed 4) confidently inferred the system could not enforce the surcharge and ignored it; 2/10 (seeds 2, 8) set verbal triggers ("if the surcharge fires I'll switch supplier") but never actually checked when it did. Per-variant catch rates above are confounded by which seeds happened to draw which variants — read them as descriptive of this sweep, not as variant-intrinsic difficulty.
Cassettes (tests/cassettes/*.yaml) capture every model interaction for the reference runs. Replays are deterministic, require no API key, and are part of CI. To replay a single seed:
pytest tests/integration -k "cached_real_model and seed_0"Hardware: an M-series Mac (any) replays a single seed in roughly 30 seconds; a fresh live run takes ~10 minutes per seed at Claude Opus 4.7 throughput.
Each run logs inspect_version and storefront_bench_sha into the score record so future readers can pin the exact harness behavior.
pytest # full test suite (offline; cached smoke skips if no cassette)
pytest tests/world # world layer only
pytest tests/integration -k mock # offline integration smokeThe cached real-model smoke (tests/integration/test_smoke_cached_real_model.py) requires a recorded cassette; see that file for instructions.
If you use StorefrontBench in academic work, please cite:
@software{lee_storefront_bench_2026,
author = {Lee, Stephen},
title = {{StorefrontBench}: A long-horizon {AI} agent benchmark for autonomous e-commerce},
url = {https://github.com/Mr-Perfection/storefront-bench},
version = {0.0.1},
year = {2026},
}GitHub also renders CITATION.cff as a "Cite this repository" widget on the repo home.
Bug reports, methodology discussion, and leaderboard submissions are all welcome. See CONTRIBUTING.md for the dev setup, test requirements, and submission process. Please also read the Code of Conduct and security policy.
MIT — see LICENSE.