Community-driven benchmarking of 22 NVIDIA NIM models — fully automated, zero infra cost, self-hostable in minutes.
🚀 View Live Dashboard · 📖 Docs · 🤝 Contribute · 💬 Discussions
NIMStats automatically benchmarks 22 NVIDIA NIM models every hour using GitHub Actions and publishes the results to a beautiful, interactive dashboard. No servers, no subscriptions — just fork, add your API key, and go.
| 🏎️ Hourly Benchmarks | 📊 Interactive Charts | 🔁 Zero Infrastructure | 🌍 Fully Open-Source |
|---|---|---|---|
| Automatic via GitHub Actions | Response time, throughput & trends | Static site + free CI/CD | Fork and self-host in minutes |
Get your own benchmarking dashboard running in under 5 minutes.
git clone https://github.com/MauroDruwel/NIMStats.git
cd NIMStatsVisit build.nvidia.com → Create a free account → Copy your API key.
In your forked repo: Settings → Secrets and variables → Actions → New repository secret
| Name | Value |
|---|---|
NIM_API_KEY |
Your NVIDIA NIM API key |
| Platform | Steps |
|---|---|
| Cloudflare Pages | Connect repo → auto-deploys on every push to main |
| GitHub Pages | Settings → Pages → Deploy from main |
| Netlify / Vercel | Connect repo for instant auto-deploy |
Actions → Benchmark NVIDIA NIM Models → Run workflow
That's it — your dashboard auto-refreshes every hour. ✨
| Tab | What you get |
|---|---|
| 📊 Overview | 5 animated KPI cards · success trend charts · top-10 speed & throughput bars · model reliability pills |
| 🏆 Leaderboard | Composite score rankings · sortable columns · SVG sparklines · trend indicators (↑↓→) · provider chips |
| 🔬 Explorer | Per-model deep dive · response time history chart · error breakdown donut · availability heatmap |
| ⏱ Timeline | Filterable run history (All / 24h / 48h / 7d) · expandable run cards with full per-model detail |
| ⚔️ Compare | Head-to-head overlay chart · win-rate stats · side-by-side metric comparison |
| 🔗 Public APIs | Multiple category endpoints: /top (balanced), /top/speed (speed & tps), and /top/intelligence (capabilities) in both JSON and raw .txt formats. Perfect for integration with local scripts, scripts, or apps |
NIMStats exposes lightweight, static API endpoints for querying the #1 model in different performance categories. Every time the hourly benchmark completes, these endpoints are updated.
| Category | Endpoint (JSON) | Endpoint (Plain Text) | Scoring Balance |
|---|---|---|---|
| ⚖️ Balanced (Overall) | /top |
/top/model |
30% Uptime + 30% Intelligence + 20% Avg Time + 20% Throughput |
| 🏎️ Speed & Throughput | /top/speed |
/top/speed/model |
50% Avg Response Time + 50% Throughput (TPS) |
| 🧠 Model Intelligence | /top/intelligence |
/top/intelligence/model |
70% Artificial Analysis Score + 30% Uptime |
{
"best_model": "nvidia/nemotron-3-nano-omni-30b-a3b-reasoning",
"provider": "nvidia",
"score": 71,
"intelligence": 14.9,
"uptime": 90.6,
"avg_response_time_ms": 4736.7,
"best_response_time_ms": 432.0,
"avg_time_to_first_token_ms": 312.5,
"avg_throughput_tps": 163.3,
"total_runs": 720,
"success_count": 652,
"wins": 364,
"last_seen": "2026-07-07T10:00:08Z",
"generated_at": "2026-07-07T10:08:49Z"
}22 models across 11 providers — click to expand
| Provider | Model | Highlight |
|---|---|---|
| DeepSeek | deepseek-ai/deepseek-v4-flash |
Fast MoE, optimized for speed |
| DeepSeek | deepseek-ai/deepseek-v4-pro |
Professional-grade reasoning |
| Z-AI | z-ai/glm-5.2 |
Superior code understanding |
| MiniMax | minimaxai/minimax-m2.7 |
Efficient inference model |
| MiniMax | minimaxai/minimax-m3 |
Latest MiniMax generation |
| NVIDIA | nvidia/nemotron-3-super-120b-a12b |
NVIDIA's 120B flagship |
| NVIDIA | nvidia/nemotron-3-nano-omni-30b-a3b-reasoning |
Compact omni reasoning model |
| NVIDIA | nvidia/llama-3.3-nemotron-super-49b-v1.5 |
Nemotron Super 49B v1.5 |
| Moonshot | moonshotai/kimi-k2.6 |
Context-optimized model |
| OpenAI | openai/gpt-oss-120b |
Open-source 120B |
google/gemma-4-31b-it |
Lightweight edge inference | |
| Qwen | qwen/qwen3.5-397b-a17b |
Flagship Qwen (397B) |
| Qwen | qwen/qwen3.5-122b-a10b |
Mid-range Qwen 3.5 MoE |
| Qwen | qwen/qwen3-next-80b-a3b-instruct |
Next-gen Qwen (80B MoE) |
| Mistral | mistralai/mistral-large-3-675b-instruct-2512 |
Largest Mistral (675B) |
| Mistral | mistralai/mistral-medium-3.5-128b |
Efficient medium-scale Mistral |
| Mistral | mistralai/mistral-small-4-119b-2603 |
Mistral Small 4 (119B) |
| Meta | meta/llama-3.3-70b-instruct |
Llama 3.3 70B |
| Meta | meta/llama-4-maverick-17b-128e-instruct |
Llama 4 Maverick (128 experts) |
| Meta | meta/llama-3.2-90b-vision-instruct |
Multimodal 90B vision model |
| StepFun | stepfun-ai/step-3.5-flash |
Ultra-fast flash model |
| StepFun | stepfun-ai/step-3.7-flash |
Latest high-performance flash |
┌──────────────────── GitHub Actions (every hour) ──────────────────────┐
│ │
│ ┌─────────────────────┐ ┌─────────────────────┐ │
│ │ Job 1 — Group A │ │ Job 2 — Group B │ (parallel) │
│ │ N/2 NIM models │ │ N/2 NIM models │ │
│ └──────────┬──────────┘ └──────────┬──────────┘ │
│ └──────────────┬───────────────┘ │
│ ┌────────▼────────┐ │
│ │ Merge + commit │ → history.db committed to repo │
│ └─────────────────┘ │
└───────────────────────────────────────────────────────────────────────┘
│
┌──────────▼───────────┐
│ Cloudflare Pages │ → auto-deploys on push
│ (static dashboard) │ index.html + history.db
└──────────────────────┘
Parallel jobs = ~50% faster benchmarks ⚡
Change the benchmark prompt
Edit PROMPT in scripts/test_models.py:
PROMPT = "Your custom prompt here"Add or remove models
Use the model management script:
# List models in DB vs test_models.py
python scripts/manage_models.py list
# Add a new model to ALL_MODELS
python scripts/manage_models.py add your/custom-model
# Remove a model from ALL_MODELS and purge its data from history.db
python scripts/manage_models.py remove your/custom-model
# Purge all DB models not in ALL_MODELS
python scripts/manage_models.py purgeOr manually edit ALL_MODELS in scripts/test_models.py:
ALL_MODELS = [
"your/custom-model",
# ...
]Change the schedule
Edit .github/workflows/benchmark.yml:
- cron: '0 */6 * * *' # Every 6 hours instead of every hourRun locally
# Serve the dashboard
python3 -m http.server 8000
# Open http://localhost:8000
# Run benchmarks manually (requires NIM_API_KEY env var)
export NIM_API_KEY=your_key_here
python3 scripts/test_models.pyhistory.db is a SQLite database persisted in the repo — the single source of truth. The browser loads it via sql.js (WebAssembly) and queries it entirely client-side. scripts/results.json is a temporary per-job artifact that is never committed.
Schema Architecture:
CREATE TABLE prompts (
id INTEGER PRIMARY KEY,
text TEXT UNIQUE
);
CREATE TABLE models (
id INTEGER PRIMARY KEY,
name TEXT UNIQUE,
intelligence_score REAL DEFAULT NULL
);
CREATE TABLE errors (
id INTEGER PRIMARY KEY,
text TEXT UNIQUE
);
CREATE TABLE runs (
id INTEGER PRIMARY KEY,
timestamp TEXT NOT NULL,
prompt_id INTEGER REFERENCES prompts(id),
fastest_model_id INTEGER REFERENCES models(id),
fastest_time INTEGER
);
CREATE TABLE model_results (
run_id INTEGER REFERENCES runs(id),
model_id INTEGER REFERENCES models(id),
success INTEGER NOT NULL,
error_id INTEGER REFERENCES errors(id),
response_time INTEGER,
tokens_generated INTEGER,
total_tokens INTEGER,
time_to_first_token INTEGER,
PRIMARY KEY (run_id, model_id)
);Benchmark parameters: temperature: 0.7 · top_p: 0.9 · max_tokens: 500 · OpenAI-compatible API
Contributions are what make the open-source community amazing. Any contribution you make is greatly appreciated!
- Fork the repository
- Create your feature branch:
git checkout -b feat/amazing-feature - Commit your changes:
git commit -m 'feat: add amazing feature' - Push to the branch:
git push origin feat/amazing-feature - Open a Pull Request
Ideas for contributions:
- 🆕 Add new NIM models to the benchmark list
- 📊 New chart types or dashboard widgets
- 🌐 Internationalization / translations
- 🐛 Bug fixes and performance improvements
- 📖 Improve documentation
Please read through open Issues before starting — someone might already be working on it!
- NVIDIA NIM API Documentation
- NVIDIA Model Catalog
- GitHub Actions Docs
- sql.js — SQLite in the browser
Distributed under the MIT License. See LICENSE for details.
Made with ❤️ for the ML community · ⭐ Star this repo if you find it useful!