OptaneMoE-Stream is an experimental Windows runtime and deployment toolkit for running Mixture-of-Experts models larger than GPU memory. It keeps cold expert weights on independent Intel Optane Persistent Memory DAX volumes, prefetches routed experts through pinned host memory, and overlaps storage, Host-to-Device transfer, and GPU computation.
Quick start · Results · Architecture · Compatibility · Reproduction guide · 中文简介
Important
This is a research preview, not a transparent RAM replacement. It currently targets compatible GGUF v3 MoE models and a pinned llama.cpp b10173-derived Windows runtime. Read the compatibility and safety sections before writing large expert packs.
Large sparse MoE models may have hundreds of gigabytes of total weights while activating only a small expert subset for each token. Loading the complete model into a 24 GiB GPU is impossible, but loading only the routed experts can be practical—if storage latency is hidden behind useful computation.
Optane App Direct is a particularly interesting tier for this job: it is byte-addressable, persistent, and can be exposed as independent NTFS-DAX volumes on Windows. OptaneMoE-Stream preserves those independent DAX volumes; it does not combine them with RAID or Storage Spaces.
The original GGUF remains read-only. The deployment tool identifies safely
sliceable expert tensors, distributes complete (layer, expert) bundles across
one or more targets, writes temporary packs, verifies every byte by SHA-256,
and commits packs only after all targets pass verification.
Test system: Windows 11 Pro, Intel Xeon Platinum 8480+ engineering sample, RTX 4090 24 GiB, Intel Optane Persistent Memory 300 in App Direct/DAX mode. Model: DeepSeek-V4-Flash-0731 UD-IQ3_XXS. Each group used three fresh-process runs with a fixed prompt, seed, context, and generation length.
| Expert path | Load time | Prompt tok/s | Decode tok/s | Decode CV |
|---|---|---|---|---|
| Vanilla CPU-MoE / P4800X | 43.217 s | 1.435 | 1.681 | 31.42% |
| P4800X → pinned DDR → GPU slots | 45.843 s | 0.737 | 1.180 | 4.34% |
| Single DAX → pinned DDR → GPU slots | 44.492 s | 1.281 | 1.574 | 0.43% |
| Dual DAX → readers → pinned DDR → GPU slots | 40.763 s | 1.284 | 1.569 | 2.54% |
Dual DAX improved decode throughput by 32.9% over the staged P4800X path, but was statistically tied with single DAX for batch-1 decode. Separate probes showed why: dual DAX reached about 18.31 GiB/s into pinned DDR, while the end-to-end dual-DAX-to-RTX-4090 pipeline reached about 12.39 GiB/s. The second device adds host-read bandwidth, but sparse jobs, prediction misses, synchronization, and the shared PCIe/GPU copy path limit steady-state scaling.
The complete data, raw sanitized logs, methodology, and limitations are in
data/ and docs/. Historical values
without retained raw console logs are explicitly marked unverified.
OptaneMoE-Stream is currently installed from source. The Python deployment
tools use only the standard library; no pip install step is required.
git clone https://github.com/soxmonitor/OptaneMoE-Stream.git
cd OptaneMoE-Stream
# Verify the downloaded project and its sanitized experiment data.
python -m unittest discover -s code\tests -v
.\scripts\validate-package.ps1
# Check Git, Python, CMake, CUDA visibility, and optionally prepare/build
# the pinned llama.cpp runtime. This writes only under .\vendor and .\build.
.\scripts\install.ps1 -CheckOnly
.\scripts\install.ps1 -PrepareRuntime -BuildRuntime -Jobs 16install.ps1 refuses to replace a dirty or incorrectly pinned llama.cpp tree.
It does not configure firmware, create PMem regions, format disks, or modify a
DAX volume. Increase -Jobs to match the build capacity of your workstation.
After installation, use the model deployment planner below. Always start with
-PlanOnly because expert packs may consume tens or hundreds of gigabytes.
- Windows 11 or Windows Server with Intel Optane PMem in App Direct mode;
- one or more NTFS-DAX volumes already provisioned and verified;
- Python 3.10 or newer;
- a CUDA-capable GPU and compatible llama.cpp build environment;
- enough free target capacity for the expert payload plus a 1 GiB reserve.
The normal deployment path never formats a disk and never modifies the source GGUF. Provisioning PMem regions and DAX filesystems is intentionally outside the one-command model deployment flow.
.\scripts\deploy-model.ps1 `
-FirstShard D:\Models\model-00001-of-00004.gguf `
-Target P0=P:\OptaneMoE\model\experts-p.pack, `
Q0=Q:\OptaneMoE\model\experts-q.pack `
-Slug model-optane `
-DisplayName "Model (Optane P/Q)" `
-Description "Local MoE with Optane expert streaming" `
-PlanOnlyReview the generated layout summary, model architecture, expert roles, target sizes, and free-space requirements before continuing.
Run the same command without -PlanOnly. Existing pack or .building paths
are refused rather than overwritten. Pack writers run in parallel across target
volumes and every completed pack is re-read and verified.
The research patch is preserved under
code/llama.cpp-patch/. Apply it to the pinned upstream
revision documented in THIRD_PARTY_NOTICES.md, then
build llama.cpp with CUDA on Windows. Prebuilt private runtimes and model weights
are deliberately not distributed in this repository.
For the fully controlled benchmark procedure, follow
docs/05-reproduction.md.
The current planner supports single-file and split GGUF v3 models when:
- expert tensors use canonical llama.cpp names;
- the expert axis is the final, contiguous, quantization-block-aligned axis;
- expert roles are
gate/up/down, fusedgate_up/down, orup/down; - the pinned runtime already supports the model architecture and quantization.
Unsupported or non-contiguous layouts fail before a pack is written. Therefore the first release promises compatible GGUF MoE models, not every MoE model from every framework. Architecture adapters and a public compatibility matrix are on the roadmap.
- Source GGUF shards are opened read-only.
- Pack targets are never silently overwritten.
- Capacity is checked before large writes begin.
- Temporary packs are atomically committed only after verification.
- Runtime offsets are validated against mapped pack sizes.
- Each DAX volume remains independent; no RAID or Storage Spaces is created.
- Models, credentials, private prompts, machine IDs, and absolute user paths are excluded from the repository.
This project is experimental systems software. Keep backups and test with non-critical data first. See SECURITY.md for responsible reports.
The pack planner and multi-pack runtime path are functional and backed by
sanitized experiment data. The next performance work is measurement-first:
per-pack queue instrumentation, persistent readers, coalesced sparse reads,
route-aware ub4 waves, bounded speculation, and dependency-scoped overlap.
See ROADMAP.md for release stages. DSA, IAA, AMX offload, automatic firmware configuration, and destructive disk provisioning are not first-release features.
code/tools/ GGUF inspection, placement, packing, and benchmarks
code/llama.cpp-patch/ pinned research runtime source and upstream license
code/dax-cuda-pipeline-probe DAX → pinned DDR → CUDA measurement probe
code/tests/ placement and index tests
data/raw-sanitized/ sanitized machine-readable evidence
data/summaries/ compact result tables and counters
docs/ methodology, results, reproduction, limitations
scripts/ package validation and release archive helpers
.github/ Windows CI and contribution templates
Hardware reports, compatibility fixtures, parser adapters, Windows build improvements, and reproducible performance traces are welcome. Please read CONTRIBUTING.md before opening a pull request. Performance claims must include correctness checks, raw counters, run conditions, repeated runs, and variance.
Citation metadata is provided in CITATION.cff. Original
project material is MIT-licensed. The modified llama.cpp file retains its
upstream MIT notice; see THIRD_PARTY_NOTICES.md.
Model weights are not included and remain subject to their own licenses.
OptaneMoE-Stream 是一个面向 Windows 的实验性 MoE 专家流式加载项目。它把 超出显存容量的冷专家权重保存在独立的傲腾 App Direct / NTFS-DAX 卷上，只将 当前路由命中的专家通过 pinned DDR 异步传入 GPU，并尝试与上一层计算重叠。
当前版本支持符合 llama.cpp 常见 GGUF 专家张量布局的模型，不宣称无条件支持
所有 MoE。部署前应先运行 --plan-only；程序不会自动格式化傲腾盘，也不会把
多个 DAX 卷组成 RAID。完整实验过程、原始脱敏数据、局限和复现步骤均已随仓库
公开。
Maintained by 飯野龙马 · Research preview · Windows App Direct / DAX