Afterimage

Run BF16 models too big for your GPU, at full precision.

Quick start · Results · Cross-model benchmark · Architecture · Configuration · Research

Afterimage compresses your model's weights losslessly and streams them through your GPU a layer at a time. A 29.5 GB model runs on an 8 GB card, bit-for-bit identical to the original. No quantization, no accuracy loss.

Turn on speculative decoding with a small draft model and it's 3x faster than streaming alone, and faster than Hugging Face's own Accelerate offload at the same memory. Every number below is a real run on real hardware, not a projection.

It comes with a CLI, a web UI, an OpenAI-compatible server, and a Python API. There's also an opt-in research lab where nineteen speedup ideas get tested against named controls and reported honestly, wins and losses both. If the full model already fits in your GPU, a normal in-memory engine will be faster; Afterimage is for when it doesn't.

Results

Qwen3-14B (29.536 GB BF16), RTX 3080 Laptop GPU (8 GB), WSL2/CUDA, cold page cache, four prompt families × four forced greedy tokens:

Configuration	Peak VRAM	Seconds/token	vs AirLLM	Exactness
Afterimage + fixed speculation	3.813 GB	9.150	3.15x	Greedy-token exact at T=0
Hugging Face Accelerate GPU/CPU/disk	3.800 GB	14.318	2.02x	Same BF16 checkpoint and token IDs
Afterimage exact + 4 GB residency	3.934 GB	17.360	1.66x	Reference-execution equivalent
AirLLM 3.1.0	1.583 GB	28.861	1.00x	Same BF16 checkpoint and token IDs
Afterimage chunked output head	0.901 GB	29.606	0.97x	Approximate BF16 matmul
Afterimage exact minimum-memory	1.723 GB	32.514	0.89x	Reference-execution equivalent

The honest reading:

• At the exact low-memory floor, AirLLM wins.
• At about 4 GB without speculation, Hugging Face Accelerate wins.
• With fixed speculation, Afterimage wins this suite: 1.56x Accelerate and 3.15x AirLLM at 3.813 GB.
• The 0.901 GB Afterimage point is not lossless execution. Blocking the output head changes BF16 reduction order even when the final token happens to agree.

See all H0-H18 hypotheses, controls, results, external comparisons, and the ranked conclusion. The raw JSON is in results/.

The broader cross-family and scale campaign adds Phi-4 Mini 3.8B and Mistral Small 24B. It finds a real Pareto boundary: Accelerate is fastest on both new checkpoints; AirLLM owns the exact low-VRAM 24B point; and Afterimage certified MIPS reaches 27.539 s/token at 2.915 GB, 1.66x AirLLM but 9.52% behind Accelerate. Across Phi, Qwen, and Mistral, the lossless store remains stable at 1.45–1.49x compression.

Why Afterimage

• Run the original checkpoint. The measured 29.5 GB BF16 model runs on an 8 GB GPU without weight quantization.
• Choose the tradeoff explicitly. Set a VRAM and optional host-RAM budget; infeasible exact plans fail before generation instead of silently degrading.
• Store and stream fewer bytes. The Qwen3-14B store is 20.328 GB, a 1.453x lossless reduction from 29.536 GB.
• Use spare memory productively. Residency eliminates repeated reads; speculation amortizes one streamed target pass over several committed tokens.
• Keep contracts visible. Exact, greedy-token-exact, distribution-exact, and approximate modes are labeled in configuration and benchmark output.
• Use it as a service. The package exposes one-shot generation, job control, an OpenAI-compatible endpoint, a web UI, and machine-readable experiment runs.

Quick start

git clone https://github.com/iodriller/Afterimage.git
cd Afterimage
./start

That's the whole thing, on macOS, Linux, and WSL2. First run sets everything up and runs a small model end to end so you can see it work. Every run after that just starts the server. On Windows, double-click start.bat instead (or run .\install.ps1).

Once it's running:

afterimage compress Qwen/Qwen3-14B
afterimage run Qwen/Qwen3-14B \
  "Explain why the sky appears blue in two sentences." \
  --vram-budget-gb 4 --stats

Prefer Docker?

docker compose up

Docker GPU execution requires the NVIDIA Container Toolkit.

Add speculative decoding

afterimage run Qwen/Qwen3-14B \
  "Write a short Python function that checks whether a number is prime." \
  --vram-budget-gb 4 \
  --draft-model Qwen/Qwen3-0.6B \
  --spec-k 8 --stats

At temperature zero, speculative decoding commits the same greedy tokens as the target. At nonzero temperature it samples from the target distribution; the draft changes efficiency, not the target distribution.

Python API

from pathlib import Path

from transformers import AutoTokenizer

from afterimage.runtime.config import EngineConfig
from afterimage.runtime.streaming_engine import (
    StreamingLosslessModel,
    load_draft_model,
)

config = EngineConfig(
    vram_budget_gb=4.0,
    ram_budget_gb=8.0,
    draft_mode="model",
    spec_k=8,
)

model = StreamingLosslessModel(
    "Qwen/Qwen3-14B",
    store_dir=Path.home() / ".afterimage/stores/Qwen__Qwen3-14B",
    device="cuda",
    config=config,
)
draft = load_draft_model("Qwen/Qwen3-0.6B", device="cuda")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-14B")
input_ids = tokenizer("Explain entropy in one paragraph.", return_tensors="pt").input_ids.cuda()
tokens, policy = model.generate_adaptive(
    input_ids,
    max_new_tokens=64,
    draft_model=draft,
)

Server

pip install -e ".[server]"
afterimage serve --host 127.0.0.1 --port 8420

The server exposes /v1/chat/completions, compression job controls, pause/resume/cancel, budget feasibility, runtime statistics, and the H0-H18 Experiment Lab.

Research lab

Research methods are opt-in configurations and never replace the stable runtime defaults automatically.

afterimage research experiments --json
afterimage research test-plan h9-ram-overlay-head --json
afterimage research pin-preflight --gigabytes 0.35 --json
afterimage research profile-trace TRACE.json --out PROFILE.json
afterimage research optimize-residency TRACE.json \
  --manifest STORE/manifest.json --vram-budget-gb 4 --out PLAN.json

Current research summary:

• H9 is the strongest positive mechanism screen, but only at 0.6B on this WSL2 host because the 14B head exceeds its pinned-memory ceiling.
• H1 is the strongest positive live 14B candidate, at +1.61% versus control.
• H6 predicts a 38.56% preparation reduction and now has a scalable 441-tensor planner, but still needs held-out live execution.
• H16 and H17 regressed. H18's exact KV rollback passed its mechanism gate but stopped for L2 futility (-0.59% paired median; 90% interval -4.62% to +1.09%).
• The remaining candidates are below gate, action-identical, or contradicted.

No H1-H18 candidate has L3 confirmatory superiority evidence. Fixed speculation is a stable core configuration, not one of the failed adaptive candidates, and is the web UI's default profile.

Documentation

Document	Purpose
All hypotheses and baselines	Controlling results table, rankings, AirLLM/Accelerate comparisons, novelty assessment
Architecture	Runtime, storage, memory-tier, speculation, and evidence diagrams
Configuration	Stable profiles and advanced flags
How it works	Implementation walkthrough and AirLLM contrast
Research methods	H0-H18 definitions, controls, metrics, and kill gates
Hypothesis lineage	Literature source and novelty boundary for each idea
Results log	Chronological corrections and raw-run interpretation
Contributing	Development and verification workflow

Status and limits

• Benchmarks are single-machine research evidence, not universal performance claims. Effects below about 10% should be treated cautiously without paired confirmation.
• Exact low-VRAM generation is slow: a streamed 14B target reads or reconstructs most weights for every target pass.
• WSL2 limits available pinned system memory. H9's full 14B test needs native Linux or another host that can genuinely pin at least 1.6 GB.
• When the full model fits in GPU memory, use an in-memory engine such as vLLM, TensorRT-LLM, or Transformers instead.
• macOS runs CPU-only today; there's no CUDA, so the GPU decode kernels don't run. A streamed 14B model on CPU is slow enough to be a demo, not a daily tool. Apple Silicon's unified memory often means the model you want fits directly anyway, which is the case Afterimage doesn't need to solve.

Apache-2.0. Contributions and reproducible counter-results are welcome.