Update dependency cache_dit to v1.5.2

[renovate]

ℹ️ Note

This PR body was truncated due to platform limits.

This PR contains the following updates:

Package	Change	Age	Confidence
cache_dit	==1.1.10 → ==1.5.2

---

Release Notes

vipshop/cache-dit (cache_dit)

v1.5.2

Compare Source

Release Notes: cache-dit v1.5.2

Release Date: 2026-09-07
Comparison: v1.5.1...v1.5.2 (5 commits)
Type: Hotfix release

Summary

v1.5.2 is a hotfix release focused on stability of the caching + distributed-parallelism
combination. It fixes a crash when --cache is combined with context parallelism
(--parallel ulysses/ring), a NameError in the image_reward metric, a layerwise-offload
correctness bug for models with tied (weight-shared) parameters, and a cache-setup failure for
multi-transformer pipelines. It also adds an extra_parallel_kwargs escape hatch for
model-specific context-parallel options and fixes async-Ulysses wrapper gating.

Affected Components

Component	Path	Change
Caching (DBCache blocks)	src/cache_dit/caching/cache_blocks/pattern_base.py	Fn-residual sharding under CP (fix #​1094)
Caching (adapter)	src/cache_dit/caching/cache_adapters/cache_adapter.py	Per-instance _is_cached check for multi-transformer pipes
Distributed (config/dispatch)	src/cache_dit/distributed/config.py, .../dispatch.py	extra_parallel_kwargs passthrough for transformer & controlnet
Async Ulysses	src/cache_dit/distributed/async_ulysses/*	Gate wrappers on ulysses_async; flux2 comm ops as eager boundaries for torch.compile
Layerwise offload	src/cache_dit/offload/layerwise.py	Dedupe tied weights across offload targets
Metrics CLI	src/cache_dit/metrics/metrics.py	Fix image_reward NameError
Tests	tests/api/, tests/quantization/	Regression tests for multi-transformer cache & tied-weight offload

Breaking Changes

None.

New Features

• extra_parallel_kwargs for model-specific context-parallel options (#​1110):
  ParallelismConfig now accepts an extra_parallel_kwargs dict (distinct from the deprecated
  parallel_kwargs, which maps to standard config fields). Entries are opaque to cache-dit and
  forwarded to _ContextParallelConfig.extra_kwargs on both transformer and controlnet dispatch,
  letting downstream model hooks read custom toggles

Bug Fixes

• Fix crash when --cache is combined with --parallel ulysses/ring (#​1113, fixes #​1094):
  the cache Fn-blocks residual (_get_Fn_residual) subtracted a full-sequence
  original_hidden_states from the already context-parallel-sharded hidden_states, raising a
  shape-mismatch error (e.g. 1280 vs 640, ratio == CP size). The CP-enabled guard only
  inspected diffusers' native _diffusers_hook registry, while cache-dit's own Ulysses/Ring
  runtime attaches split hooks to the vendored _cache_dit_hook registry, so the guard was always
  False for the built-in CP backend. The fix detects split hooks in both registries and
  re-shards the full-sequence tensor via the block's own split hook before subtracting (with an
  explicit shape guard and a safe proxy fallback). The non-CP path is unchanged.
  Verified on Wan2.2-T2V: --cache --parallel ulysses|ring runs with real cache hits (~2x
  speedup at 49 steps) and structurally-identical output.

• Fix image_reward metric raising NameError (#​1112): cache-dit-metrics image_reward ...
  always died with NameError: name 'compute_reward_score' is not defined because the call site
  unpacked the directory-level compute_reward_score which was never imported (only the unused
  compute_reward_score_img was bound). The correct helper is now imported and bound, matching
  the working clip_score sibling.

• Fix layerwise offload corruption for tied (weight-shared) parameters (#​1111): weight tying
  (e.g. lm_head.weight sharing embed_tokens.weight, Qwen3-4B) made two offload targets reference
  the same live parameter with independent pinned mirrors and onload/offload state machines — one
  target's CPU rebind poisoned the other's GPU residency bookkeeping, and async prefetch exposed
  half-copied tensors (torn embedding reads / device-mismatch crashes). Shared states are now
  owned by their first target (duplicates dropped as bookkeeping-only transports), owners are
  forced persistent, and prefetch residency bytes are recomputed after dedupe. Verified
  bitwise-equal to the GPU-resident baseline across sync / async / async+prefetch.

• Fix cache setup failure for multi-transformer pipelines (#​1102): CachedAdapter.create_context()
  set _is_cached on the pipe class while _context_manager is set on the instance. For
  dual/multi-DiT models that call enable_cache once per transformer over fresh
  FakeDiffusionPipeline instances sharing one class, the first setup flipped the class-level flag
  and every sibling was then reported as already cached (without its own context manager),
  asserting in create_context(). Context creation is now skipped only when that specific instance
  actually has a context manager. A regression test was added.

• Fix async-Ulysses wrapper gating (part of #​1110): all async Ulysses processor wrappers
  (flux, flux2, qwen_image, zimage, zimage_controlnet, ovis_image, longcat_image) entered the async
  path on ulysses_degree > 1 alone. Because the class-level patch is process-wide, a second
  instance of the same processor with CP enabled but ulysses_async=False (e.g. klein-ht and
  klein-fit in one worker) was forced onto the async path once any instance opted in. Wrappers now
  also require ulysses_async=True, restoring per-instance opt-in.

• Keep flux2 async comm ops as eager boundaries for torch.compile (part of #​1110): inline
  _All2AllComm send/wait() calls induced graph breaks that pushed surrounding fp8
  torchao Linear calls into nested resume frames, crashing dynamo ("Polyfill handler does not
  have a traceable function"). The comm operations are now kept as eager boundaries.

Performance Improvements

• No standalone performance work in this release; the cache + Ulysses/Ring combination restored by
  #​1113 delivers the expected DBCache speedups under context parallelism (~2x measured on
  Wan2.2-T2V at 49 steps).

Documentation

• No documentation changes in this release.

Dependencies

• No dependency version changes.

Testing

• New regression tests: tests/api/test_multi_transformer_is_cached.py (multi-transformer cache
  setup), tests/quantization/test_layerwise_offload.py (tied-weight offload parity).
• End-to-end verification on Wan2.2-T2V (A14B): --cache with --parallel ulysses and ring
  across 2 GPUs; cache-hit correctness and structural (non-garbled) output confirmed; non-CP path
  byte-for-byte unchanged.

Contributors

Thank you to everyone who contributed to v1.5.2:

• DefTruth (@​DefTruth)
• Tai An / Anai-Guo (@​Anai-Guo)
• jiangyon.ren (@​jiangyon-amd)
• copilot-swe-agent[bot]

Full Changelog

v1.5.1...v1.5.2

Pull requests: #​1113, #​1112, #​1111, #​1110, #​1102 · Issue fixed: #​1094

v1.5.1

Compare Source

🚀 Cache-DiT v1.5.1 Release Notes

Release Date: 2026-09-01
Comparison: v1.5.0...v1.5.1 (45 commits, 43 PRs, 7 contributors)

📋 Overview

Cache-DiT v1.5.1 is a feature release spanning ~2.5 months (2026-06-16 ~ 2026-09-01), covering 43 PRs. The headline feature is the FFPA CUDA attention backends (fp16/fp8/fp4) for Blackwell GPUs with native zero-copy NHD layouts. The release also adds 7 new model integrations (boogu-image, ernie-image, Joy-Image-Edit, AnyFlow & AnyFlow-FAR, Krea-2, Bria-Fibo, Cosmos3 Omni), a Sage3 Blackwell attention backend, torchao NVFP4 dynamic quantization, an alternating SVD refinement option for SVDQuant, the hyper-parameter-free FoCa calibrator, Context-Parallelism communication optimization for boogu-image, and Ascend NPU profiling support.

✨ Core Highlights

1. ⚡ FFPA CUDA Attention Backends

New attention backends built on ffpa-attn (Flash Prefill Attention), delivering quantized CuTe TMA kernels for sm_120 (Blackwell) GPUs — forward inference only, Context Parallelism compatible:

• --attn ffpa: CuTe TMA fp16/bf16 kernels
• --attn ffpa_fp8: FP8 quantized attention, defaults to the highest-precision quant config (int8 QK MMA + fp16 PV accumulation, per-thread Q/K + per-channel V scales on consumer GPUs)
• --attn ffpa_fp4: NVFP4 quantized attention
• --attn ffpa_fp8_per_block: per-block quantized FP8 variant (fastest, lower precision)

Fine-grained CLI controls: --ffpa-hybrid (hybrid fp16 early-rows stage, default off for every backend), --ffpa-hybrid-n-early, --ffpa-fp8-hadamard / --ffpa-fp4-hadamard (Hadamard pre-rotation). The enable_fp8 + enable_fp4 combination is explicitly rejected with a clear error.

Native NHD layouts (#​1104–#​1107): all CuTe families (fp16/fp8/fp4) now consume NHD ([B, H, N, D]) inputs natively — both packed and strided — through the fp8 persist-D tensor_layout fast path, eliminating the BHSD↔NHD permute + .contiguous() materialization overhead on every forward.

Performance (FLUX.1-dev, 2048×2048, single RTX PRO 5000, 28 steps; E2E latency / PSNR / SSIM vs BF16 baseline, measured before the NHD fast path landed):

Backend	Latency	Speedup	PSNR / SSIM
BF16 baseline	92.46s	1.00×	—
sage	81.09s	1.14×	40.2 dB / 0.986
sage3	80.73s	1.15×	31.4 dB / 0.869
ffpa (fp16)	93.24s	~1.00×	49.3 dB / 0.997
ffpa_fp8	83.64s	1.11×	32.7 dB / 0.919
ffpa_fp4	78.66s	1.17×	33.6 dB / 0.949

With 2×GPU Ulysses Context Parallelism: ffpa_fp8 48.42s, ffpa_fp4 45.89s. The fp16 row reflects the pre-NHD permute overhead, which the native NHD path (#​1104–#​1107) removes.

PRs: #​1098, #​1099, #​1100, #​1101, #​1103, #​1104, #​1105, #​1106, #​1107

2. 🧩 New Model Integrations

Seven new pipelines gain Cache + Parallelism support (cache adapters, CP/TP wiring, and cache_dit.generate CLI examples):

• boogu-image (#​1072) — plus a dedicated CP optimization: reduced KV all-2-all communication overhead for boogu (#​1082) and per-model attention backend dispatch (#​1083)
• ernie-image (#​1073)
• Joy-Image-Edit (#​1079)
• AnyFlow & AnyFlow-FAR (#​1081)
• Krea-2 (#​1074)
• Bria-Fibo (#​1075)
• Cosmos3 (Omni) (#​1088, community contribution by @​johnnynunez) — new Cosmos3OmniPatchFunctor (functor_cosmos3.py) reorders the MoT dual-stream contract so DBCache tracks the actually-denoised gen stream (und is quasi-static and must not drive residual-diff decisions). Measured on Cosmos3-Nano T2V (RTX PRO 6000 Blackwell, 189 frames / 35 steps): 480p 138.59s → 76.75s (1.81×) with DBCache rdt=0.20, 2.04× with torch.compile; 256p 1.53×. Requires diffusers main for Cosmos3OmniPipeline.

3. 🔥 Sage3 Blackwell Attention Backend

• --attn sage3 (#​1085): SageAttention-3 backend for sm_120 (Blackwell) GPUs
• Fixed the sage3 (sm_120) tensor layout (#​1086)
• Made sage3-attn compatible with torch.compile (#​1087)

4. 💎 Quantization Enhancements

• torchao NVFP4 dynamic quantization (#​1090): new quant types nvfp4 and nvfp4_weight_only on the torchao backend for Blackwell GPUs. torchao>=0.17.0 is now required for the quantization extra.
• Alternating SVD refinement for SVDQuant (#​1095, community contribution by @​Rudin6): new opt-in svd_refine_iters knob. Each round refits the low-rank factors against smoothed_weight - dequant(quantize(residual)) so the low-rank branch absorbs the residual quantizer's error pattern. Default 0 is bitwise-identical to the previous one-shot behaviour (verified across 72 configurations). Measured on 14 PixArt-Sigma layers with real calibration activations: 5 rounds cut weight error by 7.8–8.9% and layer output error by 17.4–18.2% for both INT4 and NVFP4.
• SVDQ W4A4 pad-to-128 (#​1084): pad SVDQ linear layers to a multiple of 128 so the W4A4 kernel covers shapes that are not tile-aligned.
• Fix: clamp torchao float8 zero scales to avoid 0/0 NaN (#​1097) — previously, all-zero scale rows produced NaNs on dequantization.
• Cleaner quantization logging format (#​1091)

5. 🔮 FoCa Calibrator

New FoCaCalibrator (#​1066): forecasts cached tensors with a BDF2 predictor + Heun corrector. It is a drop-in alternative to TaylorSeerCalibrator and DMDCalibrator with zero hyper-parameters — step intervals are derived automatically from the step counters.

6. 📈 Parallelism & Profiling

• CP communication optimization (#​1082): reduced boogu-image KV all-2-all communication overhead on the Context-Parallelism path.
• Ascend NPU profiling (#​1096, with @​changetheway and @​ooooooye): cache_dit.profiler now supports torch_npu.profiler for Ascend NPUs, mirroring the torch.profiler API.

🔧 Other Enhancements

• CLI: support extra input kwargs (#​1068); allow using local test data (#​1092); new cache+dmd+svdq YAML example configs (#​1069)
• Docs: Cache-DiT tech report added (#​1062); DMD calibrator design docs (#​1063, #​1064, #​1065); docs rendering fix (#​1067); SVDQ NVFP4 kernel comments (#​1071)
• Community: new community integration links in README (#​1108)
• Dev experience: agent workflows + model-integration skill updates (#​1074, #​1075, #​1076, #​1080); skills relocated to .github/skills (#​1109); example data updates (#​1077, #​1078)

🐛 Bug Fixes

• sage3 (sm_120) attention backend tensor layout (#​1086)
• torchao float8 zero scales clamped to avoid 0/0 NaN (#​1097)
• ffpa_fp8_per_block V quantization switched to per-channel for 2048-resolution precision (#​1103)

⬆️ Dependencies

• torchao>=0.14.1 → torchao>=0.17.0 (quantization extra; required for NVFP4 dynamic quantization)

👥 Contributors

Thanks to everyone who contributed to this release:

@​DefTruth, @​Rudin6 (first contribution), @​johnnynunez (first contribution), @​changetheway, @​ooooooye, Copilot

Full Changelog

v1.5.0: Major Release

Compare Source

🚀 Cache-DiT v1.5.0 Release Notes

Release Date: 2026-06-16
Comparison: v1.3.0...v1.5.0 (176 commits)
Full Changelog: GitHub Releases

📋 Overview

Cache-DiT v1.5.0 is a major feature release spanning 3 months (2026-03-12 ~ 2026-06-16), covering 176 PRs. This release is organized around four core modules: SVDQuant W4A4 Quantization (PTQ/DQ/NVFP4/Converter CLI), DMD Calibrator (exponential-basis Dynamic Mode Decomposition), Bucket-style Layerwise CPU Offload (compute-communication overlapped offloading), and Ray Wrapper (transparent distributed inference). Additional highlights include FLUX.2-klein-kv series support, full CUDA Graph integration, extensive kernel refactoring, quantization enhancements, parallelism framework improvements, and comprehensively updated documentation.

✨ Core Highlights

1. 💎 SVDQuant W4A4 Quantization

Cache-DiT v1.5.0 natively integrates a complete SVDQuant W4A4 quantization workflow — the most significant feature of this release. Instead of relying on third-party libraries, users can now perform end-to-end W4A4 quantization directly through Cache-DiT's cache_dit.quantize() / cache_dit.load() API.

📊 PTQ (Post-Training Quantization)

Supports svdq_int4_r{rank} and svdq_nvfp4_r{rank} quant types:

• INT4 PTQ (≥sm80): Collect activation statistics via calibrate_fn → SVD low-rank decomposition → INT4 packing → runtime W4A4 GEMM. Supports three calibration precision levels: low (recommended default, ~18× speedup), medium, high. Serialize to {quant_type}.safetensors + quant_config.json; restore via cache_dit.load().
• NVFP4 PTQ (≥sm120, Blackwell): Designed for RTX 5090 and other Blackwell GPUs. Currently only runtime_kernel="v1" is supported for NVFP4.

Performance (FLUX.2-klein-4B, 1024×1024, L20):

Stage	Latency (s)	Memory (GiB)	Transformer Weight (GiB)
BF16 baseline	2.13	17.32	7.22
SVDQuant INT4	1.24	12.39	2.28
SVDQuant + compile	1.02	12.39	2.28

• Transformer weight reduction: ~3.2× compression (7.22 → 2.28 GiB)
• End-to-end latency: ~1.7× speedup (2.13 → 1.24s), ~2.1× with compile (2.13 → 1.02s)
• PSNR > 29 dB, near-lossless visual quality

NVFP4 Performance (RTX 5090, FLUX.2-klein-4B, 1024×1024):

Stage	Latency (s)	Speedup	Memory (GiB)
BF16 baseline	0.97	1.00×	17.32
NVFP4 PTQ	0.58	1.69×	12.50
NVFP4 + compile	0.47	2.05×	12.50

⚡ DQ (Dynamic Quantization)

Zero-calibration quantization via _dq suffix types (e.g., svdq_int4_r128_dq):

• identity (default): Apply SVD low-rank decomposition directly to the original weight matrix — no calibration, no serialization.
• weight / weight_inv: Weight-statistics-only heuristic smooth strategies (experimental).
• few_shot: Collect a small number of real inference forwards at runtime, then quantize in-place with configurable relaxation strategies (7 strategies: auto/stable_auto/power/log/rank/top/fixed). Supports few_shot_auto_compile for deferred compilation after quantization.

DQ Performance (FLUX.2-klein-4B, 1024×1024, identity, rank=128): 1.28s, PSNR 28.71 dB.

🔧 SVDQ Converter CLI

New cache-dit-convert command-line tool for one-click model conversion to SVDQ W4A4:

cache-dit-convert --model-path black-forest-labs/FLUX.2-klein-4B \
  --save-dir ./FLUX.2-klein-4B-svdq \
  --quant-type svdq-int4-r128-dq

Supports INT4/NVFP4 DQ conversion, custom smooth strategies, and multiple precision options. Outputs {quant_type}.safetensors + quant_config.json.

🔀 Fused MLP

New fused_gelu_mlp / fused_gelu_proj passes (enable via svdq_kwargs["fused_mlp"]=True), fusing the first GEMM + GELU activation into a single kernel for reduced kernel launch overhead and intermediate activation memory.

🔗 Parallelism Compatibility (Cache-DiT Exclusive)

SVDQuant in Cache-DiT fully supports Context Parallelism (Ulysses / Ring / USP) and Tensor Parallelism (PyTorch DTensor). Users can layer quantization on top of distributed parallelism for extreme memory compression and throughput gains. SVDQW4A4ShardLinear (dtensor.py) provides native TP sharding support. This is a differentiating capability of Cache-DiT's SVDQuant implementation versus other W4A4 solutions.

⚙️ Quantization Configuration Enhancements

• Regional Quantization (regional_quantize=True + repeated_blocks): Quantize only repeated transformer blocks, keeping embedding layers at full precision.
• Hybrid Precision Plan (precision_plan): Assign different quant types to different sub-layers by name pattern.
• FP8 Per-Tensor Fallback (per_tensor_fallback=True, default on): Auto-fallback from per-row to per-tensor for TP-incompatible layers.
• TorchAO Backend Refactor: Cleaner QuantizeBackend enum (AUTO / TORCHAO / CACHE_DIT / NONE).
• Quantize API Refactor: Deprecated legacy kwargs, unified under QuantizeConfig + svdq_kwargs.

📦 cache-dit-cu13 Pre-built Wheel

Pre-compiled SVDQuant wheel for CUDA 13 users: pip install cache-dit-cu13 — no source build needed.

---

2. 💾 Bucket-style Layerwise CPU Offload

Cache-DiT v1.5.0 introduces a novel bucket-style layerwise offload mechanism that overcomes the "every layer waits for its own transfer" inefficiency of traditional approaches.

Core Design:

• Bucket Pipeline: Divide target modules into small contiguous buckets; prefetch the next bucket asynchronously while the current one executes.
• Dual Independent Copy Stream Pools: Separate CUDA stream pools for onload (H2D) and offload (D2H).
• Persistent Bins: Distribute the persistent budget evenly across the target sequence.
• Flexible Resource Controls: transfer_buckets, persistent_buckets, persistent_bins, prefetch_limit, max_copy_streams, max_inflight_prefetch_bytes.

Performance (FLUX.1-dev, L20):

Config	Memory	Latency
No offload	~38 GiB	23.4s
Diffusers sequential	~1 GiB	335s
Layerwise (transfer=4, persistent=32, bins=4)	~16 GiB	24.6s

Only ~1.2s added latency versus no-offload baseline for a 38→16 GiB memory reduction.

torch.compile Compatible: Apply offload before compile; offload hooks work correctly after compilation.

CLI Quick Start:

python3 -m cache_dit.generate flux \
  --layerwise-offload --layerwise-async-transfer \
  --layerwise-transfer-buckets 4 --layerwise-persistent-buckets 32 \
  --layerwise-persistent-bins 4 --layerwise-prefetch-limit \
  --layerwise-max-inflight-prefetch-bytes 8gib --compile

---

3. 🌩️ Ray Wrapper (Transparent Distributed Inference)

The Ray Wrapper makes distributed inference completely transparent to user code. No torchrun, no dist.init_process_group, no manual model sharding — just use_ray=True, and Cache-DiT handles everything.

Two Wrapper Levels:

Level	Description	Best For
Pipeline Wrapper (recommended)	Ray manages the entire pipeline execution	Full feature support (cache, quant, parallelism), simplest, fastest.
Transformer Wrapper	Only the transformer runs on Ray workers	Lightweight, but slight slower

Key Features:

• ray_transfer_fn: User-defined per-worker model loading, bypassing serialization overhead and custom module class resolution issues.
• ray_use_compile: Automatic per-worker compilation.
• ray_runtime_env: Custom module import handling via PYTHONPATH.
• Supports all parallelism strategies: TP, Ulysses, Ring.
• LoRA support: fuse before enabling (TP requires fused LoRAs).

Performance (FLUX.2-klein-base-9B):

Config	Latency
Baseline (single GPU)	47.41s
Ray TP=2 + compile	24.57s

Minimal Example:

cache_dit.enable_cache(
  pipe,
  parallelism_config=ParallelismConfig(tp_size=2, use_ray=True),
)
image = pipe(prompt="A cat...").images[0]  # Code unchanged

---

4. 🔮 DMD Calibrator (Dynamic Mode Decomposition)

DMD (Dynamic Mode Decomposition) is an exponential-basis feed-forward calibrator, serving as a drop-in alternative to TaylorSeer's polynomial basis.

Mathematical Principle: The cached feature stream is modeled as a linear dynamical system $Y_{t+1} \approx A \cdot Y_t$. The propagator $A$ is identified via one economy SVD, then eigendecomposed for extrapolation via $\lambda^k$. Unlike TaylorSeer's polynomial extrapolation (diverges as $t^n \to \infty$), DMD is bounded when $\lvert\lambda\rvert \leq 1$.

TaylorSeer vs DMD:

Aspect	TaylorSeer (Polynomial)	DMD (Exponential)
Basis	$Y(t) \approx \sum \frac{d^nY}{dt^n} \frac{t^n}{n!}$	$Y(t) \approx \Phi \cdot \text{diag}(\lambda^t) \cdot b$
Extrapolation	Diverges as $t^n \to \infty$	Bounded when $\lvert\lambda\rvert \leq 1$
Snapshots needed	2+ (1st order)	≥ 4 uniformly spaced
Best for	DiT-class denoising (DDPM)	Flow-matching generators (Hunyuan3D, etc.)
Noise sensitivity	Low	Moderate (SVD truncation suppresses noise)

Usage:

cache_dit.enable_cache(
  pipe,
  cache_config=DBCacheConfig(...),
  calibrator_config=DMDCalibratorConfig(
    dmd_history=6, dmd_rank=0, dmd_ridge=1e-8,
  ),
)

CLI: python -m cache_dit.generate flux --cache --dmd --dmd-history 6

---

🔧 Other Enhancements

🧩 FLUX.2-klein-kv Series

• TP + compile integration (#​888)
• fp8 per-row + TP support (#​896)
• Async Ulysses support (#​877)

📈 CUDA Graph

• Full CUDA Graph support (#​942-#​952)
• CUDA Graph + fp8 rowwise integration (#​952)
• descent_tuning enabled by default (#​935)
• compile full-graph CLI option (#​946)

🏗️ Kernel & Infrastructure Refactoring

• Triton kernel refactoring (3/N, #​907-#​909)
• Communication kernels registered as torch.library ops (#​905)
• CuTeDSL communication kernels (fp8, #​977), merge-attn-states kernel (#​973)
• Unified all2all/ring communication API (#​985)
• Async Ulysses refactoring (#​986)
• Unified ops registration policy (#​931, #​939, #​967)

🔗 Parallelism Improvements

• record_plans for TP/TE-P planners (#​916-#​917)
• sub cp_plan support (#​989)
• CP & VAE-P planner refactoring for better logging (#​918)

⚡ Compilation Optimizations

• Removed manual graph breaks (#​885)
• CUDA Graph for dynamic compile (#​951)
• SVDQuant + compile compatibility; offload + compile (#​1014)
• Log suppression improvements (#​934, #​923)

🤝 Community Integrations

• MindIE-SD: Huawei Ascend NPU attention/compilation backend support (@​blian6, #​1004)
• TensorRT-LLM community link (#​991)

📦 Dependency Updates

• PyTorch → 2.11.0 (#​900)
• uv for dependency management (#​992)
• Python version compatibility fix (@​FNGarvin, #​1025)
• TorchAO ≥ 0.17.0

📝 Documentation

• New/reworked docs: QUANTIZATION.md, OFFLOAD.md, RAY.md, CACHE_API.md (DMD), CUDA_GRAPH.md, COMPILE.md
• Formatting and typo fixes (#​886-#​887, #​899, #​903, #​913)
• FAQ update: Flash Attention 2 install guide (#​915)
• Community link fixes (#​928, #​940)

⚠️ Breaking Changes

Change	PR	Migration
Serving module deprecated	#​933	Migrate to SGLang Diffusion or vLLM-Omni
Native Diffusers parallelism backend deprecated	#​1017	Use Cache-DiT native parallelism backends (better performance)
Quantize API legacy kwargs deprecated	#​910-#​911	Migrate to unified QuantizeConfig + svdq_kwargs. Legacy grad_ckpt, reorder_before_quantize etc. removed

👥 New Contributors

Thank you to the new contributors who joined the Cache-DiT community:

• @​blian6 — MindIE-SD NPU attention/compilation backend support
• @​FNGarvin — Python version compatibility fix
• @​Archerkattri — DMD Calibrator

🙏 Contributors

Thank you to all contributors for this release: @​DefTruth, @​Archerkattri, @​FNGarvin, @​blian6.

For the full list of changes, see GitHub Release v1.5.0 and the full changelog.

---

🇨🇳 中文版

📋 概述

Cache-DiT v1.5.0 是一次重大功能更新，历时 3 个月（2026-03-12 ~ 2026-06-16），涵盖 176 个 PR。本次发布围绕四大核心模块：SVDQuant W4A4 量化（PTQ/DQ/NVFP4/转换器CLI）、DMD Calibrator（基于指数基底的动态模态分解校准器）、Bucket-style Layerwise CPU Offload（计算-通信重叠的逐层卸载）、Ray Wrapper（分布式推理透明化包装器）。此外还包括 FLUX.2-klein-kv 系列支持、CUDA Graph 完整集成、大量 kernel 重构与量化增强、并行框架改进、以及全面更新的文档体系。

---

✨ 核心亮点

1. 💎 SVDQuant W4A4 量化

Cache-DiT v1.5.0 原生集成了完整的 SVDQuant W4A4 量化工作流，这是本次发布最重要的特性。与其依赖第三方库，用户现在可以直接通过 Cache-DiT 的 cache_dit.quantize() / cache_dit.load() API 完成从校准到推理的全链路 W4A4 量化。

📊 PTQ（后训练量化）

支持 svdq_int4_r{rank} 和 svdq_nvfp4_r{rank} 两种量化类型：

• INT4 PTQ（≥sm80）：通过 calibrate_fn 收集激活统计 → SVD 低秩分解 → INT4 打包 → 运行时 W4A4 GEMM。支持三种校准精度：low（推荐默认，~18x 加速）、medium、high。序列化到 {quant_type}.safetensors + quant_config.json，通过 cache_dit.load() 一键恢复。
• NVFP4 PTQ（≥sm120，Blackwell）：专为 RTX 5090 等 Blackwell GPU 设计，使用 NVFP4 格式打包权重，当前仅支持 runtime_kernel="v1"。

性能数据（FLUX.2-klein-4B，1024×1024，L20）：

阶段	延迟 (s)	显存 (GiB)	Transformer 权重 (GiB)
BF16 baseline	2.13	17.32	7.22
SVDQuant INT4	1.24	12.39	2.28
SVDQuant + compile	1.02	12.39	2.28

• Transformer 权重：~3.2× 压缩（7.22 → 2.28 GiB）
• 端到端延迟：~1.7× 加速（2.13 → 1.24s），compile 叠加后 ~2.1× 加速（2.13 → 1.02s）
• PSNR 保持 29+ dB，视觉质量几乎无损

NVFP4 性能（RTX 5090，FLUX.2-klein-4B，1024×1024）：

阶段	延迟 (s)	加速比	显存 (GiB)
BF16 baseline	0.97	1.00×	17.32
NVFP4 PTQ	0.58	1.69×	12.50
NVFP4 + compile	0.47	2.05×	12.50

⚡ DQ（动态量化）

无需校准数据的零样本量化，类型后缀 _dq（如 svdq_int4_r128_dq）：

• identity（默认）：直接对原始权重矩阵做 SVD 低秩分解，无需校准、无需序列化
• weight / weight_inv：仅基于权重统计量的启发式平滑策略（实验性）
• few_shot：运行时收集少量前向的激活统计后实时量化，支持 7 种松弛策略（auto/stable_auto/power/log/rank/top/fixed），可配置 few_shot_steps、few_shot_relax_factor、few_shot_relax_top_ratio、few_shot_relax_strategy。支持 few_shot_auto_compile 在量化完成后自动触发 torch.compile

DQ 性能（FLUX.2-klein-4B，1024×1024，identity smooth，rank=128）：1.28s，PSNR 28.71 dB。

🔧 SVDQ Converter CLI

新增 cache-dit-convert 命令行工具，一键将预训练模型转换为 SVDQ W4A4 格式：

cache-dit-convert --model-path black-forest-labs/FLUX.2-klein-4B \
  --save-dir ./FLUX.2-klein-4B-svdq \
  --quant-type svdq-int4-r128-dq

支持 INT4/NVFP4 DQ 转换、自定义 smooth 策略、多种精度选项。生成 {quant_type}.safetensors + quant_config.json，可通过 cache_dit.load() 直接加载。

🔀 Fused MLP

新增 fused_gelu_mlp / fused_gelu_proj pass（通过 svdq_kwargs["fused_mlp"]=True 启用），将第一个 GEMM + GELU 激活融合为单 kernel，降低 kernel launch 开销和中间激活显存占用。

🔗 与并行兼容（Cache-DiT 独家特色）

SVDQuant 在 Cache-DiT 中完整支持 Context Parallelism（Ulysses / Ring / USP）和 Tensor Parallelism（PyTorch DTensor）。用户可以在分布式推理场景下叠加使用量化 + 并行，实现极致的显存压缩和吞吐提升。SVDQW4A4ShardLinear（dtensor.py）提供原生 TP 分片支持。这是 Cache-DiT 中 SVDQuant 区别于其他 W4A4 实现的差异化能力。

⚙️ 量化配置增强

• 区域量化（regional_quantize=True + repeated_blocks）：仅量化 transformer 的重复块，保持嵌入层等敏感层全精度
• 混合精度计划（precision_plan）：按层名模式为不同子层指定不同量化类型（如 attn.to_q 用 float8_per_tensor、attn.to_k 用 float8_weight_only）
• FP8 Per-Tensor Fallback（per_tensor_fallback=True，默认开启）：在 TP 场景下，不支持 per-row 量化的层自动回退到 per-tensor，消除跳过警告，提升覆盖率（144 → 144，0 skip）
• TorchAO 后端重构：更清晰的 QuantizeBackend 枚举（AUTO / TORCHAO / CACHE_DIT / NONE）
• 量化 API 重构：废弃旧版 kwargs，统一为 QuantizeConfig + svdq_kwargs

📦 cache-dit-cu13 预编译 Wheel

为 CUDA 13 用户提供预编译的 SVDQuant wheel：pip install cache-dit-cu13，免去从源码编译 SVDQ kernel 的麻烦。

---

2. 💾 Bucket-style Layerwise CPU Offload

Cache-DiT v1.5.0 引入了全新的 bucket 式逐层卸载机制，解决了传统逐层 offload "每层等传输"的低效问题。

核心设计：

• Bucket Pipeline：将目标模块切分为连续小桶，当前桶执行时异步预取下个桶，实现计算-通信重叠
• 双独立 Copy Stream 池：onload（H2D）和 offload（D2H）各自拥有独立 CUDA stream 池
• Persistent Bins：将常驻预算均匀分布到目标序列上，避免热权重集中在 prefix
• 灵活的资源控制：transfer_buckets（预取深度）、persistent_buckets（常驻桶数）、persistent_bins（常驻分布桶数）、prefetch_limit（保守预取限制）、max_copy_streams（并发拷贝流数）、max_inflight_prefetch_bytes（预取字节预算）

性能数据（FLUX.1-dev，L20）：

配置	显存	延迟
无 offload	~38 GiB	23.4s
原生 diffusers offload	~25 GiB	56s
原生 diffusers sequential	~1 GiB	335s
Layerwise（transfer=4, persistent=32, bins=4）	~16 GiB	24.6s

仅 1.2s 的额外延迟（vs 无 offload 23.4s），即可将显存从 38 GiB 压缩到 16 GiB。

兼容 torch.compile：先应用 offload 再编译，offload hooks 在编译后正常工作。

CLI 快速启动：

python3 -m cache_dit.generate flux \
  --layerwise-offload --layerwise-async-transfer \
  --layerwise-transfer-buckets 4 --layerwise-persistent-buckets 32 \
  --layerwise-persistent-bins 4 --layerwise-prefetch-limit \
  --layerwise-max-inflight-prefetch-bytes 8gib --compile

3. 🌩️ Ray Wrapper（分布式推理透明化）

Ray Wrapper 让分布式推理对用户代码完全透明。你不需要 torchrun、不需要 dist.init_process_group、不需要手动模型分片 —— 只需 use_ray=True，Cache-DiT 接管一切。

两种包装级别：

级别	描述	推荐场景
Pipeline Wrapper（推荐）	Ray 管理整个 pipeline 执行，包括 text encoder、VAE、scheduler	完整功能支持（cache、量化、并行），最简单的用户体验
Transformer Wrapper	仅 transformer 由 Ray 执行，其他组件留在主进程	轻量级，但比pipeline level 慢

核心特性：

• ray_transfer_fn：用户自定义每个 worker 的模型加载逻辑，绕过序列化/反序列化开销，解决自定义模块的类解析问题
• ray_use_compile：worker 内自动编译
• ray_runtime_env：通过 PYTHONPATH 处理自定义模块导入
• 支持 TP、Ulysses、Ring 等所有并行策略
• LoRA 支持：建议融合后使用（TP 不支持未融合 LoRA）

性能数据（FLUX.2-klein-base-9B）：

配置	延迟
Baseline（单卡）	47.41s
Ray Wrapper TP=2 + compile	24.57s

最简用法：

cache_dit.enable_cache(
  pipe,
  parallelism_config=ParallelismConfig(tp_size=2, use_ray=True),
)
image = pipe(prompt="A cat...").images[0]  # 代码完全不变

---

4. 🔮 DMD Calibrator（动态模态分解校准器）

DMD（Dynamic Mode Decomposition）是一个基于指数基底的前馈校准器，作为 TaylorSeer（多项式基底）的替代方案。

数学原理：将缓存特征流建模为线性动力系统 $Y_{t+1} \approx A \cdot Y_t$，通过一次 SVD 识别传播子 $A$，特征分解后以 $\lambda^k$ 进行外推。相比 TaylorSeer 的多项式外推（$t^n$ 发散），DMD 在 $\lvert\lambda\rvert \leq 1$ 时有界。

TaylorSeer vs DMD 对比：

维度	TaylorSeer（多项式）	DMD（指数）
基底	$Y(t) \approx \sum \frac{d^nY}{dt^n} \frac{t^n}{n!}$	$Y(t) \approx \Phi \cdot \text{diag}(\lambda^t) \cdot b$
外推行为	$t^n \to \infty$ 发散	$\lvert\lambda\rvert \leq 1$ 时有界
快照要求	2+（一阶）	≥ 4 均匀间隔
最佳场景	DiT 类去噪（DDPM）	流匹配生成器（Hunyuan3D 等）
噪声敏感性	低	中等（SVD 截断抗噪）

使用方式：

cache_dit.enable_cache(
  pipe,
  cache_config=DBCacheConfig(...),
  calibrator_config=DMDCalibratorConfig(
    dmd_history=6, dmd_rank=0, dmd_ridge=1e-8,
  ),
)

CLI：python -m cache_dit.generate flux --cache --dmd --dmd-history 6

---

🔧 其他增强

🧩 FLUX.2-klein-kv 系列支持

• TP + compile 集成（#​888）
• fp8 per-row + TP 支持（#​896）
• Async Ulysses 支持（#​877）

📈 CUDA Graph

• 完整 CUDA Graph 支持（#​942-#​952）
• CUDA Graph + fp8 rowwise 联合使用（#​952）
• descent_tuning 默认启用（#​935）
• compile full-graph CLI option（#​946）

🏗️ Kernel 与基础设施重构

• Triton kernel 重构（3/N，#​907-#​909）
• 通信 kernel 注册为 torch.library ops（#​905）
• CuTeDSL 通信 kernel（fp8，#​977）、merge-attn-states kernel（#​973）
• 统一 all2all/ring 通信 API（#​985）
• 异步 Ulysses 重构（#​986）
• ops 注册策略统一（#​931, #​939, #​967）

🔗 并行改进

• record_plans 函数支持（TP/TE-P planners，#​916-#​917）
• sub cp_plan 支持（#​989）
• CP & VAE-P planner 重构以改进日志（#​918）

⚡ 编译优化

• 移除手动 graph break（#​885）
• CUDA Graph for dynamic compile（#​951）
• SVDQuant + compile 兼容（#​1014 offload + compile）
• 日志抑制优化（#​934, #​923）

🤝 社区集成

• MindIE-SD：华为昇腾 NPU 的注意力/编译后端支持（@​blian6，#​1004）
• TensorRT-LLM 社区链接（#​991）

📦 依赖升级

• PyTorch → 2.11.0（#​900）
• 使用 uv 替代 pip 管理依赖（#​992）
• Python 版本兼容性修复（@​FNGarvin，#​1025）
• TorchAO ≥ 0.17.0

📝 文档

• 新增/大幅更新文档：QUANTIZATION.md、OFFLOAD.md、RAY.md、CACHE_API.md（DMD）、CUDA_GRAPH.md、COMPILE.md
• 文档格式化与 typo 修复（#​886-#​887, #​899, #​903, #​913）
• FAQ 更新：Flash Attention 2 安装指南（#​915）
• 社区链接修复（#​928, #​940）

⚠️ Breaking Changes

变更	PR	迁移指引
Serving 模块废弃	#​933	推荐迁移至 SGLang Diffusion 或 vLLM-Omni
Native Diffusers 并行后端废弃	#​1017	使用 Cache-DiT 原生并行后端（性能更好）
Quantize API 废弃旧 kwargs	#​910-#​911	统一使用 QuantizeConfig + svdq_kwargs，旧版 grad_ckpt、reorder_before_quantize 等参数已移除

👥 新贡献者

感谢以下新贡献者加入 Cache-DiT 社区：

• @​blian6 — MindIE-SD NPU 注意力/编译后端支持
• @​FNGarvin — Python 版本兼容性修复
• @​Archerkattri — DMD Calibrator

🙏 致谢

感谢所有贡献者：@​DefTruth、@Archerkattri、@FNGarvin、@blian6。

完整变更列表请参见 GitHub Release v1.5.0。

v1.3.12

Compare Source

What's Changed

• chore: update installation guide by @​DefTruth in #​1035
• chore: Update README.md by @​DefTruth in #​1036
• Update README.md by @​DefTruth in #​1037
• chore: update installation guide by @​DefTruth in #​1038
• feat: config yaml support svdq dq/few-shot by @​DefTruth in #​1040
• [2/N]: config yaml support svdq dq/few-shot by @​DefTruth in #​1042
• chore: update why not svqd nvfp4 for sm100 by @​DefTruth in #​1043
• chore: hightlight bucket-style layerwise offload by @​DefTruth in #​1044
• chore: Update README.md by @​DefTruth in #​1045
• svdq: add fused gelu mlp/proj pass by @​DefTruth in #​1047
• docs: add fused gelu mlp/proj docs by @​DefTruth in #​1048

Full Changelog: vipshop/cache-dit@v1.3.11...v1.3.12

v1.3.11

Compare Source

What's Changed

• chore: update release tools by @​DefTruth in #​1034

Full Changelog: vipshop/cache-dit@v1.3.10...v1.3.11

v1.3.10

Compare Source

What's Changed

• feat: add MindIE-SD as optional NPU attention and compilation backend by @​blian6 in #​1004
• chore: simplify attn backend auto select by @​DefTruth in #​1024
• Fix Python version mismatch in setup.py by @​FNGarvin in #​1025
• offload: extract copy stream pool and split init by @​DefTruth in #​1026
• feat: support svdquant nvfp4 ptq/dq by @​DefTruth in #​1029
• chore: Update README.md by @​DefTruth in #​1030
• whl: cache-dit-cu13 pkg w/ svdq kernels by @​DefTruth in #​1031
• whl: fix build_releases.sh tool by @​DefTruth in #​1032
• whl: fix build_releases.sh tool by [@&#8203

✂ Note

PR body was truncated to here.

---

Configuration

📅 Schedule: (UTC)

• Branch creation
  • At any time (no schedule defined)
• Automerge
  • At any time (no schedule defined)

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

---

• If you want to rebase/retry this PR, check this box

---

This PR was generated by Mend Renovate. View the repository job log.