CloudAI Autotune

CloudAI Autotune is a lightweight experiment manager for LLM serving benchmarks. It sits on top of NVIDIA CloudAI: CloudAI runs the benchmark, while Autotune records what was tried, parses the result, stores the metrics, and recommends the next config value to test.

It is designed for the common tuning loop:

try a config -> measure throughput/latency -> compare history -> choose next config

Example:

Run 1: batch_size = 1   ->  120 tok/s,  90 ms latency
Run 2: batch_size = 4   ->  330 tok/s, 160 ms latency
Run 3: batch_size = 8   ->  430 tok/s, 260 ms latency

Recommendation: try batch_size = 6 because 8 crossed the latency budget and 6
has not been tested yet.

What Autotune Is

Autotune is:

• a CLI for running or ingesting CloudAI benchmark experiments
• a parser for JSON, JSONL, and text benchmark outputs
• a SQLite experiment database
• a simple recommender for the next knob value to try
• a Streamlit dashboard for browsing experiment history

Autotune is not:

• a benchmark engine by itself
• a replacement for CloudAI
• a full multi-variable optimizer yet
• a storage trace, POSIX/S3, or checkpoint I/O benchmark tool

Architecture

flowchart TD
    A[CloudAI TOML config] --> B[autotune run]
    B --> C[CloudAI CLI]
    C --> D[runs/run_id/stdout.log]
    C --> E[runs/run_id/report.json]

    F[Existing report JSON/JSONL/log] --> G[autotune ingest]

    D --> H[Parser]
    E --> H
    G --> H

    H --> I[Normalized metrics]
    I --> J[(autotune.db SQLite)]
    A --> J

    J --> K[autotune list]
    J --> L[autotune recommend]
    J --> M[Streamlit dashboard]

    L --> N[Next untested knob value]

Loading

The important boundary is that CloudAI owns benchmark execution. Autotune owns experiment tracking and recommendation.

Inputs and Outputs

Inputs

Input	Example	Used by
CloudAI config	configs/examples/vllm_baseline.toml	run, derive, ingest
Existing report	reports/examples/vllm_batch4.json	ingest, demo
Tuning knob	serving.batch_size	recommend, demo
Latency budget	--latency-budget-ms 200	recommend, demo

Outputs

Output	Example	Contents
Experiment DB	autotune.db	configs, status, metrics, report paths
Run directory	runs/0001_vllm_baseline_.../	captured CloudAI artifacts
Log file	runs/.../stdout.log	CloudAI output or failure details
Recommendation	Suggested: 6.0	next untested knob value
Dashboard	Streamlit app	tables, charts, recommendation view

Metrics

Reports are normalized into a small stable metric set:

Metric	Meaning
latency_ms	latency in milliseconds
ttft_ms	time to first token in milliseconds
throughput_tokens_per_sec	generated token throughput
runtime_sec	benchmark runtime
failure_rate	failed request ratio

The parser accepts common aliases from different report formats. For example, tokens_per_second, request_throughput, and output_throughput can all map to throughput_tokens_per_sec.

Check Pass/Fail Budgets

After recording runs, check them against simple performance budgets:

autotune check \
  --latency-budget-ms 200 \
  --ttft-budget-ms 50 \
  --min-throughput-tokens-per-sec 300 \
  --max-failure-rate 0.05

Use --strict in scripts or CI to exit non-zero if any experiment fails a budget or cannot be evaluated because a required metric is missing.

Project Layout

cloudai-autotune/
  autotune/
    cli.py               # command-line interface
    config_mutator.py    # load and derive TOML configs
    runner.py            # CloudAI subprocess wrapper
    parser.py            # report/log -> normalized metrics
    database.py          # SQLite experiment store
    recommender.py       # next-value recommendation heuristic
  configs/examples/      # sample CloudAI configs
  reports/examples/      # sample benchmark reports
  dashboard/app.py       # Streamlit dashboard
  runs/                  # captured run artifacts
  tests/                 # unit tests

Setup

python3 -m venv .venv
source .venv/bin/activate
pip install -e .

Check the CLI:

autotune --help

CloudAI is only required for real benchmark runs. The local demo works without CloudAI, GPUs, or cluster access.

Quick Demo Without CloudAI

The fastest way to see the project work is:

autotune demo

This command:

1. loads bundled sample reports from reports/examples/
2. writes them to autotune-demo.db
3. recommends a next value for serving.batch_size

Useful options:

autotune demo --db /tmp/my-demo.db
autotune demo --scenario vllm_baseline
autotune demo --knob serving.batch_size --latency-budget-ms 200

Run a Real CloudAI Scenario

When CloudAI is installed and available as cloudai:

autotune run path/to/test_scenario.toml \
  --notes "baseline before tensor-parallel change" \
  --metadata hardware.gpu=A100 \
  --metadata run.nodes=1 \
  --system-config path/to/system.toml \
  --tests-dir path/to/tests \
  --hook-dir path/to/hooks

Autotune will:

1. create a database row with status=running
2. call cloudai run --config ... --output ...
3. capture stdout/stderr under runs/<run_id>/stdout.log
4. parse report.json or a common summary artifact such as cloudai-summary.json, summary.json, results.json, metrics.json, or JSONL equivalents
5. mark the experiment completed or failed

CloudAI stdout and stderr are preserved in the run's stdout.log. Autotune also appends a diagnostic for launch failures, timeouts, non-zero exits, and unreadable report artifacts. Failed runs exit non-zero so shell scripts and CI do not mistake them for successful benchmarks.

Use a custom CloudAI binary if needed:

autotune run path/to/test_scenario.toml \
  --cloudai-bin /path/to/cloudai \
  --timeout-sec 3600 \
  --system-config path/to/system.toml \
  --tests-dir path/to/tests \
  --hook-dir path/to/hooks

Use CloudAI dry-run mode to validate config wiring without launching a real benchmark:

autotune run path/to/test_scenario.toml \
  --cloudai-bin /path/to/cloudai \
  --dry-run \
  --system-config path/to/system.toml \
  --tests-dir path/to/tests \
  --hook-dir path/to/hooks

For a direct CloudAI CLI-contract smoke check without writing an experiment record:

autotune smoke-cloudai path/to/test_scenario.toml \
  --cloudai-bin /path/to/cloudai \
  --system-config path/to/system.toml \
  --tests-dir path/to/tests \
  --hook-dir path/to/hooks

Ingest Existing Reports

If a benchmark report already exists, record it without launching CloudAI:

autotune ingest reports/examples/vllm_batch4.json \
  --config configs/examples/vllm_batch4.toml \
  --notes "baseline batch size 4" \
  --metadata hardware.gpu=A100

For a first pass when you only have a report artifact, provide the scenario, backend, and any config values you want Autotune to track:

autotune ingest reports/examples/vllm_batch4.json \
  --scenario vllm_baseline \
  --backend vllm \
  --set serving.batch_size=4

Derive a New Config

Create a new config by overriding dotted TOML keys:

autotune derive configs/examples/vllm_baseline.toml configs/derived/batch8.toml \
  --set serving.batch_size=8

Then run it:

autotune run configs/derived/batch8.toml

List Experiments

autotune list

Filter by scenario:

autotune list --scenario vllm_baseline

Compare Experiments

Show config and metric differences between two recorded runs:

autotune diff 1 2

Export Results

Write experiment summaries to CSV, JSON, or Markdown for sharing in issues, pull requests, or benchmark notes:

autotune export --format csv --out reports/summary.csv
autotune export --format json --scenario vllm_baseline --out reports/vllm.json
autotune export --format markdown --out reports/summary.md
autotune export --format markdown --template issue
autotune export --format markdown --template pr

Without --out, the export prints to the terminal.

The pr template includes a Regression Check comparing the latest completed run against the completed run immediately before it — a run can regress against its immediate predecessor without being the worst run ever recorded, which the best/latest comparison alone wouldn't catch.

Get a Recommendation

autotune recommend --knob serving.batch_size --latency-budget-ms 200

The recommender compares completed experiments for one or more knobs. It tries to avoid suggesting a value that was already tested. If 4 was good and 8 crossed the latency budget, it may suggest 6 as the next untested point.

recommend accepts the same budget policy as autotune check — latency, time to first token, minimum throughput, runtime, and failure rate — so a run that regresses on any one of them is treated as over budget, not just latency:

autotune recommend \
  --knob serving.batch_size \
  --latency-budget-ms 200 \
  --ttft-budget-ms 50 \
  --min-throughput-tokens-per-sec 300 \
  --max-failure-rate 0.05

To write that suggestion directly into a new config, pass a base config and an output path:

autotune recommend \
  --knob serving.batch_size \
  --knob serving.num_requests \
  --latency-budget-ms 200 \
  --derive-from configs/examples/vllm_baseline.toml \
  --out-config configs/derived/batch6.toml

If you don't already know every knob that's been tuned, --all-knobs reports a recommendation for every numeric dotted config key seen across completed runs, instead of requiring --knob per key:

autotune recommend --all-knobs --latency-budget-ms 200

--all-knobs and --knob are mutually exclusive.

This prints one recommendation per knob and writes configs/derived/batch6.toml with the suggested values applied.

Dashboard

streamlit run dashboard/app.py

The dashboard reads the local SQLite database and shows experiment history, best/latest run comparison, a regression check against the immediately preceding run, metric charts, and the current recommendation.

Development

Run tests:

.venv/bin/python -m pytest -q

Lint and format check (matches CI):

.venv/bin/python -m ruff check autotune dashboard tests
.venv/bin/python -m ruff format --check autotune dashboard tests

Current test coverage includes:

• config derivation
• report parsing
• runner failure handling
• SQLite persistence
• CLI ingest/demo behavior
• recommendation logic

Roadmap

Goal: make Autotune the small, reliable companion for CloudAI performance tuning — easy enough for a first benchmark, useful enough for repeated production-readiness checks.

Shipped: one-command demo/ingest/run paths, cloudai-summary.json support with workload-specific fallbacks, TTFT/runtime/failure-rate budgets, explainable runs (intent, metadata, config diffs), issue/PR/benchmark export templates, local-first SQLite storage, clean non-zero-exit failure handling for a missing/failing CloudAI binary, a dashboard runtime-budget input, ruff lint/format checks in CI, run-over-run regression tracking (compare_latest_to_previous) surfaced in both the dashboard and the pr export template, and multi-knob visibility via recommend --all-knobs (discovers every numeric config key seen across completed runs, rather than requiring --knob per key up front). Kept local-first by design, not a target with an end state — every new item below should keep working with zero services.

Later — needs your input before any code gets written

• Joint multi-knob search. Actually searching combinations of knobs (not just reporting independent bests) needs a real objective function beyond the current throughput/latency ratio, a search-budget model, and a decision on whether Autotune should ever suggest untried combinations or stay strictly "next single point from history." Don't start this without agreeing on scope — it's the one item here that's a genuine design project, not a bounded fix.
• Consider wrapping CloudAI's own DSE instead of competing with it. CloudAI already has a working multi-parameter search stack — cloudai_gym.py's Gymnasium env, GridSearchAgent and reward-function agents in cloudai/configurator/, and trajectory.csv/env.csv step logs with reward per config. Before building joint search in Autotune (above), it's worth checking whether ingesting and visualizing CloudAI DSE trajectories gets most of the value with far less risk of duplicating logic CloudAI already maintains.