Super AI Desktop Benchmark is a deterministic, cross-platform benchmark for validating an AI provider used by desktop applications.
简体中文 | Specification | Result format | Security
SADB V1 answers two practical questions:
- Is the configured model and provider path stable enough to use?
- Can it reliably complete deterministic tasks commonly needed by desktop software?
It evaluates the complete service path: model, provider, network, and request configuration. It is not a psychological IQ test and does not claim to replace academic benchmarks.
| Component | Value |
|---|---|
| Protocol | sadb/1.0 |
| Question bank | sadb-bank-1.0 |
| Scorer | sadb-score-1.0 |
| Questions | 72 |
| Bank SHA-256 | 7133690d67ef91609aaf1810a1d8ebc659cee1c016d381eaf388e015d29b5598 |
| License | Apache-2.0 |
Only results with the same protocol, bank, scorer, fingerprint, and request parameters are directly comparable.
The release contains both Chinese (data/question-bank.zh-CN.json) and English (data/question-bank.en.json) questions and canonical answers. They are independent language banks with the same 72 case IDs and semantics. Language affects comprehension and output, so scores must be compared as a language-sensitivity study, never merged or averaged into one capability score. Super Optimizer uses Chinese questions and Chinese reference scores only.
The reviewable banks are available in data/question-bank.zh-CN.json and data/question-bank.en.json. Regenerate them with sadb export-bank-zh and sadb export-bank-en.
Runs five identical requests and reports success rate, nearest-rank P50/P95 latency, and structured error counts. A response counts as successful only when it has an explicit 2xx status and non-empty content.
Runs 72 independent deterministic questions across seven desktop-oriented domains. Each question is worth one raw point. No model judge or partial credit is used.
Scores are displayed as raw points and bank completion, for example 55 (76.39%). The percentage only describes completion of this fixed V1 bank.
| Rank | Model | Capability | Stability |
|---|---|---|---|
| 1 | zai-org/GLM-5.2 |
55 (76.39%) | 100% |
| 2 | moonshotai/Kimi-K2.7-Code |
53 (73.61%) | 100% |
| 3 | deepseek-ai/DeepSeek-V4-Pro |
52 (72.22%) | 100% |
| 4 | deepseek-ai/DeepSeek-V4-Flash |
28 (38.89%) | 100% |
| 5 | nex-agi/Nex-N2-Pro |
22 (30.56%) | 100% |
These provider-path results were collected on 2026-09-01 with the frozen 72-question Chinese V1 bank. They are a reproducible diagnostic snapshot, not an official SADB ranking or a general-purpose model ranking: none reached the required 72/72 valid capability responses for a standard ranking run. Super Optimizer may show the Chinese values as a local diagnostic reference only. English scores are shown only in the bilingual chart and are not used by Super Optimizer.
Download the archive for your platform from GitHub Releases and verify it against SHA256SUMS.txt.
Release assets are produced for Windows x86-64, Linux x86-64, macOS Apple Silicon, and macOS Intel.
Set the API key in the process environment. Do not pass it on the command line.
Windows PowerShell:
$env:SADB_API_KEY = "your-api-key"
.\sadb.exe stability https://provider.example/v1 model-name > stability.json
.\sadb.exe capability --language zh-CN https://provider.example/v1 model-name > capability.json
.\sadb.exe capability --language en https://provider.example/v1 model-name > capability-en.json
Remove-Item Env:SADB_API_KEYLinux or macOS:
export SADB_API_KEY='your-api-key'
./sadb stability https://provider.example/v1 model-name > stability.json
./sadb capability --language zh-CN https://provider.example/v1 model-name > capability.json
./sadb capability --language en https://provider.example/v1 model-name > capability-en.json
unset SADB_API_KEYSADB accepts OpenAI-compatible chat completion endpoints. Remote providers must use HTTPS; HTTP is allowed only for loopback addresses.
By default, provider requests have no time limit. timeout_ms: 0 in result metadata means unlimited waiting. A user may still cancel the process, and explicit provider or network errors are recorded normally.
The PowerShell wrapper builds the official runner, executes one or more models, and exports redacted JSON and CSV summaries:
.\scripts\run-ai-benchmark.ps1 `
-ConfigPath C:\path\provider.txt `
-OutputDir artifacts\my-run `
-Models @('provider/model-a', 'provider/model-b')The config file format is:
baseurl=https://provider.example/v1
apikey=replace-me
Prefer the direct CLI for automation where credentials can be supplied by a secret manager. The PowerShell file format is provided for local convenience and must never be committed.
Requirements: Rust 1.96.0 or a compatible newer toolchain.
cargo build --release -p piaoye-ai-benchmark-runner
cargo test --workspace --all-targets
cargo clippy --workspace --all-targets -- -D warningsThe binary is written to target/release/sadb (sadb.exe on Windows).
CLI / GUI / host application
|
v
SADB V1 core
/ \
deterministic ProviderTransport
scoring adapter
|
v
Provider
crates/core: frozen question bank, deterministic scorer, metrics, status model, and public result types.crates/http: OpenAI-compatible HTTP transport with TLS and credential isolation.crates/runner: officialsadbcommand-line runner.crates/conformance: black-box public contract tests.
User interfaces and integrations must call the core. They must not reimplement scoring or percentile calculations.
- API keys are read from
SADB_API_KEYand are never part of serializable result types. - Raw model responses are not persisted by the standard result model.
- V1 never executes model-generated commands or tool calls.
- The question bank contains no user files, device identifiers, or personal data.
- Provider URLs containing credentials, query strings, or fragments are rejected.
See SECURITY.md for reporting vulnerabilities.
Run the official verification script on Windows:
.\scripts\verify-v1.ps1Portable verification:
cargo fmt --all -- --check
cargo test --workspace --all-targets
cargo clippy --workspace --all-targets -- -D warningsAn implementation may claim SADB V1 Conformant only when it passes the public conformance suite and preserves the frozen version tuple and question-bank fingerprint.
- Normative specification
- Result format
- Release process
- Public publishing policy
- Chinese technical whitepaper
- Chinese technical specification
- Conformance audit
- Audit reproduction guide
- English parallel bank
- Chinese/English five-model comparison
Read CONTRIBUTING.md before opening a pull request. Changes to the frozen V1 questions, answers, ordering, normalization, or scoring require a new bank or scorer version and must not silently retain the V1 fingerprint.
For project support and private inquiries, email support@pctutu.com. Do not include API keys, passwords, private prompts, or personal data.
Licensed under the Apache License 2.0. See NOTICE for attribution information. Third-party dependencies retain their own licenses.