Wallet Benchmark

Self-contained, evidence-backed comparison of Ethereum wallets — built for a wallet product manager benchmarking against rivals.

A small set of Python generators resolve a structured wallet dataset into three interactive HTML pages. Each page is fully self-contained: data is inlined, there are no external libraries and no server — just open the file in a browser.

The seed dataset originates from Walletbeat (walletbeat.eth), an open community project benchmarking Ethereum wallets. See Acknowledgements.

Pages

Page	What it shows	Generated by
benchmark/index.html	Comparison matrix — wallets × features, citation-backed cells, three-state colouring, diff / hide-unknown toggles, CSV export. Loads the wallet data dynamically from data/wallets/*.json and remembers your selection in localStorage (must be served over http)	build.py ← template.html
benchmark/privacy.html	Privacy data-flow (Sankey) — Action → Recipient → Data collected, coloured by collection policy and recipient risk	build_privacy.py ← template_privacy.html
benchmark/gaps.html	"Strengths & Gaps" report — per baseline + peer set, the metrics where rivals beat you (and where you beat all of them), with evidence drawer and Markdown / CSV export	build_gaps.py ← template_gaps.html
benchmark/quadrant.html	Positioning quadrant — plots wallets on two chosen category scores (e.g. Privacy × Security), the "magic quadrant" slide, with median/fixed split, per-wallet score breakdown, and Markdown / CSV export	build_quadrant.py ← template_quadrant.html

See benchmark/IDEAS.md for the feature backlog and rationale.

Data

data/ holds the source dataset (32 wallets), seeded from Walletbeat and extended here:

• wallets.json — the full wallet records (also split per-wallet under wallets/). Feature values use a small $ref / $call DSL (e.g. featureSupported, notSupported, named entity references) that the generators resolve into a normalized {state, detail, ref} model per cell.
• entities.json — data recipients / infrastructure providers, with type, jurisdiction, and privacy posture.
• contributors.json — who authored each wallet entry, their affiliation, and whether they hold equity.

Building

Each page is generated — edit the matching template*.html, then run the build script. Never hand-edit index.html / privacy.html / gaps.html; they are overwritten.

cd benchmark
python3 build.py          # -> index.html (matrix shell + metric catalog) + data/wallets/manifest.json
python3 build_privacy.py  # -> privacy.html (privacy Sankey)
python3 build_gaps.py     # -> gaps.html (gap report; reuses build.py's model)
python3 build_quadrant.py # -> quadrant.html (positioning quadrant; reuses build.py's model)

No dependencies beyond the Python 3 standard library.

Finding and filling data gaps

A cell renders "—" when a datapoint is genuinely missing, and "N/A" when it doesn't apply to that wallet type (e.g. product.* features on a hardware-only wallet, which intentionally omit that namespace). The two are distinct states in the model (unknown vs na).

python3 build_gaps_report.py   # -> data/gaps.json + a completeness summary (only real "—" gaps)

data/gaps.json is a per-cell worklist (walletId, path, label, …). Once a gap has been researched, record a verdict in data/gaps-proposals.json (shape documented in apply_proposals.py) and apply it:

python3 apply_proposals.py            # dry run — shows what would change
python3 apply_proposals.py --apply    # writes per-wallet JSON + regenerates data/wallets.json
python3 build.py                      # refresh the manifest

apply_proposals.py translates each verdict into the $ref / $call DSL, requires a citation for any positive claim (never writes refTodo), and keeps data/wallets.json and data/wallets/*.json in sync.

index.html no longer inlines the wallet data: build.py bakes only the metric catalog into it and refreshes data/wallets/manifest.json (the file list the page fetches). The page resolves the $ref / $call DSL in the browser at load time, so editing a data/wallets/*.json file is reflected on refresh with no rebuild — only re-run build.py when the metric catalog (M) changes or a wallet file is added/removed (to refresh the manifest). Because it uses fetch(), it must be served over http — open it via:

cd <repo root> && python3 -m http.server   # then browse to /benchmark/index.html

The other three pages stay self-contained (data inlined) and still open by double-clicking.

How it fits together

build.py owns the $ref / $call DSL resolution and exposes build_payload(), which returns the normalized {wallets, metrics, cats} model that drives the matrix. build_gaps.py imports that same payload so the matrix and gap report share one source of truth for colour polarity (which state is favorable for the user). build_privacy.py adds the entity resolver (with a humanized fallback) used for the privacy view.

Notes for contributors

• privacy.html / gaps.html / quadrant.html are self-contained (data inlined, no server, no external libraries) — keep new features the same. index.html is the exception: it fetches the data at runtime (served over http) and persists the user's selection to localStorage. Its DSL resolver is a direct port of build.py's — keep the two in sync if you change either.
• Generated files contain multibyte glyphs; search them with grep -a or rg (plain grep treats them as binary).
• Reuse the DSL resolution in build.py and the entity resolver in build_privacy.py rather than re-walking the raw JSON.

Acknowledgements

The original seed data for this benchmark comes from Walletbeat — an open community project benchmarking Ethereum wallets (walletbeat.eth, @walletbeat). This project builds on that dataset with additional metrics, evidence, and the interactive HTML views described above. Many thanks to the Walletbeat contributors for making their work openly available. Please refer to the upstream repository for its own licensing and contribution terms.

License

This project is released under the MIT License. Note that the seed data originates from Walletbeat; refer to that project for the terms governing the upstream data.