InferStack

An observable, OpenAI-compatible LLM serving platform built with vLLM, FastAPI, Prometheus, Grafana, and Docker. It targets a single workstation with two NVIDIA GPUs while keeping the boundary to a multi-node deployment explicit.

Why this project exists

Raw model serving is only part of a usable inference service. InferStack adds bounded admission control, overload behavior, streaming-safe proxying, health checks, reproducible benchmarks, and a pre-provisioned operational dashboard without reimplementing vLLM's scheduler.

flowchart LR
    Client[OpenAI client] --> Gateway[FastAPI gateway]
    Gateway --> Queue[Bounded admission]
    Queue --> vLLM[vLLM continuous batching]
    vLLM --> G0[GPU 0]
    vLLM --> G1[GPU 1]
    Gateway -. metrics .-> Prom[Prometheus]
    vLLM -. metrics .-> Prom
    GPU[DCGM exporter] -. metrics .-> Prom
    Prom --> Grafana[Grafana]

Loading

The detailed diagram and component boundaries are in Architecture.

Features

• OpenAI-compatible chat completions, completions, embeddings, and models endpoints.
• Server-sent event streaming with backpressure and correct admission-slot lifetime.
• vLLM continuous batching, PagedAttention, tensor parallelism, and prefix caching.
• Bounded in-memory queue with HTTP 429/503 overload semantics.
• Gateway, vLLM, GPU, queue, latency, TTFT, token, and KV-cache occupancy metrics.
• Auto-provisioned Grafana dashboard and 15-day Prometheus retention.
• One-command Docker Compose startup, health checks, restarts, and persistent model cache.
• Unit/integration tests, static analysis, dependency updates, and container-build CI.
• Reproducible streaming benchmark tool with raw JSON output.

Quick start (two GPUs)

Prerequisites

• Linux with Docker Engine and the Compose plugin.
• NVIDIA driver, NVIDIA Container Toolkit, and two visible GPUs.
• Enough disk for model weights. A Hugging Face token may be required for gated models.

cp .env.example .env
# Set MODEL_ID, strong PUBLIC_API_KEY/GRAFANA_ADMIN_PASSWORD, and optional HF token.
make up
docker compose ps
./scripts/smoke_test.sh

First startup can take several minutes while weights download. Endpoints:

Service	URL	Purpose
Gateway	http://localhost:8080	OpenAI-compatible API and /docs
Grafana	http://localhost:3000	InferStack dashboard
Prometheus	http://localhost:9090	Metrics queries and target health

Example request:

curl http://localhost:8080/v1/chat/completions \
  -H 'Authorization: Bearer change-me' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "default",
    "messages": [{"role": "user", "content": "Explain PagedAttention briefly."}],
    "stream": true,
    "max_tokens": 128
  }'

OpenAI's Python client works by setting base_url="http://localhost:8080/v1" and the configured API key.

Model and GPU configuration

The default is Qwen/Qwen2.5-7B-Instruct with one tensor-parallel replica across both GPUs. Set MODEL_ID to a Hugging Face model supported by the pinned vLLM release; examples include compatible Qwen and Llama-family checkpoints.

Important .env controls:

Variable	Default	Meaning
TENSOR_PARALLEL_SIZE	2	GPUs used by one model replica
GPU_MEMORY_UTILIZATION	0.90	Fraction available to weights and KV cache
MAX_MODEL_LEN	8192	Context cap; lower it if KV-cache allocation fails
MAX_CONCURRENT_REQUESTS	32	Requests admitted to vLLM
MAX_QUEUE_SIZE	128	Requests waiting at the gateway
QUEUE_TIMEOUT_SECONDS	30	Maximum admission wait

See Multi-GPU and scaling for TP=2 versus two TP=1 replicas and the limits of autoscaling on a fixed two-GPU host.

Monitoring

The dashboard includes request and token rates, p50/p95/p99 API latency, streaming TTFT, queue depth, rejection rate, vLLM running/waiting requests, KV-cache occupancy, GPU utilization, and GPU memory.

Metrics have intentionally bounded labels: prompts, model output, request IDs, and API keys are never metric labels. Streaming token counts come from vLLM; the gateway's own token counter covers non-streaming responses. Use vLLM's native counters for complete fleet-wide token accounting.

Benchmarks

See Benchmark methodology for the concurrency sweep, measurement definitions, and results table.

Development

python -m venv .venv && source .venv/bin/activate
pip install -e '.[dev,benchmark]'
make lint
make test
make validate

Tests mock the model server and require no GPU. CI runs linting, strict type checking, coverage, Compose validation, and a gateway image build.

Design trade-offs

• One gateway worker: queue state is process-local. Multiple workers would multiply limits and make queue metrics misleading.
• No Python batching: vLLM continuously batches at token granularity and has better scheduler visibility.
• In-memory queue: simple and low latency, but not durable. Inference requests are normally retried by clients; durable queues need cancellation, deadlines, and result storage.
• Compose rather than Kubernetes: appropriate for one workstation. Multi-host autoscaling is described, not falsely claimed.
• Thin proxy contract: only known OpenAI endpoints are exposed. New endpoints should be added intentionally and tested.

More detail: Architecture · Scaling · Benchmarking · Contributing

Security and production notes

Set strong secrets, bind monitoring ports to a trusted network, put TLS and rate limiting at an ingress proxy, and never commit .env. The gateway does exact bearer-token matching but is not a complete identity platform. Review model licenses and prompt-data handling before public exposure. Report security issues as described in SECURITY.md.

License

Apache License 2.0. See LICENSE.