feat(modelopt): add format-general real-quant vLLM rollout by mxinO · Pull Request #3945 · NVIDIA-NeMo/RL

mxinO · 2026-09-01T01:51:14Z

Summary

Adds format-general ModelOpt real-quant rollout to NeMo-RL.

builds the vLLM deployment configuration from the calibrated policy before generation startup
streams ModelOpt-packed weights and format-owned sidecars through the Megatron-Bridge export plan
supports per-layer and mixed schemas without format tables in NeMo-RL
resolves grouped Hugging Face MoE mappings to native per-expert refit tensors through Megatron-Bridge
uses vLLM native layerwise reload for repeated real-quant refits
removes the prior custom NVFP4 packing, scale mapping, and quantization-method path
retains one narrow vLLM 0.25.1 compatibility backport that makes shared NVFP4 MoE scale storage writable during reload
shares real-quant orchestration across GRPO and distillation while leaving fake-quant generation unchanged

A rollout format is qualified when ModelOpt can export its canonical tensors and the pinned vLLM ModelOpt loader selects the intended native method for that exact schema. NeMo-RL does not translate unsupported schemas. Depending on vLLM, they may be rejected or may select an unquantized fallback, so representative native methods must be inspected before qualification.

Dependencies

The temporary Megatron-Bridge gitlink targets the dependency branch. Release pins will be updated after both dependency APIs land.

Online refit call flow

Collective refit replaces the IPC/ZMQ transport calls with broadcast_weights_for_collective() and update_weights_from_collective(); the export and native vLLM reload paths are unchanged.

Validation

focused NeMo-RL real-quant unit suites and branch linters passed
dense W4A16 native-loader functional training completed repeated refits with finite loss and KL
matched Qwen3-8B runs completed 150 steps for BF16-attention control and MXFP8-attention treatment, both with NVFP4 W4A16 MLP
a four-node Qwen3.5 grouped-MoE W4A4 job completed two GRPO steps and repeated native vLLM refits with finite loss, KL, and reward

This remains a draft while dependency APIs land and release pins are finalized.

Qwen3-8B mixed-format qualification

The matched training comparison covers common steps 31-150:

control: BF16 attention + NVFP4 W4A16 MLP
treatment: MXFP8 attention + NVFP4 W4A16 MLP
final validation reward/accuracy: 0.2168 control, 0.2012 treatment

MXFP8 attention projection speed

GB200, vLLM 0.25.1, Qwen3-8B TP2 QKV/O shapes, measured as one CUDA graph over 36 distinct layers and 72 projections. Values above 1.0 mean MXFP8 is faster.

Decode rows `M`	MXFP8 speed vs BF16
1	0.649x
32	0.645x
128	0.666x
512	0.881x
1024	0.956x
2048	1.194x
4096	1.285x

The current FlashInfer CuTeDSL MXFP8 path is slower at the rollout decode sizes and crosses over only at prefill-scale batches.

Signed-off-by: Meng Xin <mxin@nvidia.com>

…nt-refit Signed-off-by: Meng Xin <mxin@nvidia.com> # Conflicts: # tests/unit/algorithms/test_grpo.py

Signed-off-by: Meng Xin <mxin@nvidia.com>

copy-pr-bot · 2026-09-01T01:51:17Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

Signed-off-by: Meng Xin <mxin@nvidia.com>

This reverts commit 7a06b82. Signed-off-by: Meng Xin <mxin@nvidia.com>

Signed-off-by: Meng Xin <mxin@nvidia.com>

…nto mxin/refit-export-groups Signed-off-by: Meng Xin <mxin@nvidia.com>

Signed-off-by: Meng Xin <mxin@nvidia.com>

github-actions · 2026-09-10T03:32:34Z

✅ Submodule Fast-Forward Check Results

Check based on commit: ebce1a1 (PR #3945 from mxin/general-real-quant-refit)

✅ Submodules that are properly updated:

Megatron-Bridge: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

Signed-off-by: Meng Xin <mxin@nvidia.com>

github-actions · 2026-09-10T03:47:18Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 02d35a1 (PR #3945 from mxin/general-real-quant-refit)

✅ Submodules that are properly updated:

Megatron-Bridge: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

Signed-off-by: Meng Xin <mxin@nvidia.com>

github-actions · 2026-09-10T03:58:12Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 84ef399 (PR #3945 from mxin/general-real-quant-refit)

✅ Submodules that are properly updated:

Megatron-Bridge: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

Signed-off-by: Meng Xin <mxin@nvidia.com>

github-actions · 2026-09-10T06:00:56Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 46d845c (PR #3945 from mxin/general-real-quant-refit)

✅ Submodules that are properly updated:

Megatron-Bridge: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

Signed-off-by: Meng Xin <mxin@nvidia.com>

github-actions · 2026-09-10T06:16:09Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 79954dd (PR #3945 from mxin/general-real-quant-refit)

✅ Submodules that are properly updated:

Megatron-Bridge: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

Signed-off-by: Meng Xin <mxin@nvidia.com>

github-actions · 2026-09-10T06:46:31Z

✅ Submodule Fast-Forward Check Results

Check based on commit: a883a3e (PR #3945 from mxin/general-real-quant-refit)

✅ Submodules that are properly updated:

Megatron-Bridge: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

Signed-off-by: Meng Xin <mxin@nvidia.com>

github-actions · 2026-09-10T07:05:23Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 6b169e8 (PR #3945 from mxin/general-real-quant-refit)

✅ Submodules that are properly updated:

Megatron-Bridge: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

Signed-off-by: Meng Xin <mxin@nvidia.com>

github-actions · 2026-09-10T07:43:18Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 0918406 (PR #3945 from mxin/general-real-quant-refit)

✅ Submodules that are properly updated:

Megatron-Bridge: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

Signed-off-by: Meng Xin <mxin@nvidia.com>

github-actions · 2026-09-10T07:59:46Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 7158121 (PR #3945 from mxin/general-real-quant-refit)

✅ Submodules that are properly updated:

Megatron-Bridge: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

Signed-off-by: Meng Xin <mxin@nvidia.com>

mxinO added 14 commits August 15, 2026 09:11

WIP: route real quant refit through canonical export

b778ab2

Signed-off-by: Meng Xin <mxin@nvidia.com>

WIP: remove NVFP4-only rollout inference

55982ef

Signed-off-by: Meng Xin <mxin@nvidia.com>

WIP: cover canonical real quant orchestration

d5d6c88

Signed-off-by: Meng Xin <mxin@nvidia.com>

WIP: align real quant generation tests

ecd4fca

Signed-off-by: Meng Xin <mxin@nvidia.com>

WIP: align real quant policy tests

3846584

Signed-off-by: Meng Xin <mxin@nvidia.com>

WIP: add general real quant GRPO example

881202d

Signed-off-by: Meng Xin <mxin@nvidia.com>

WIP: use native W4A16 recipe in real quant example

ce82866

Signed-off-by: Meng Xin <mxin@nvidia.com>

Remove stale quant test import

ddeaf93

Signed-off-by: Meng Xin <mxin@nvidia.com>

Remove legacy real quant rollout controls

2fff5a8

Signed-off-by: Meng Xin <mxin@nvidia.com>

Remove stale real quant type-check exclusion

542cefb

Signed-off-by: Meng Xin <mxin@nvidia.com>

Merge remote-tracking branch 'origin/main' into mxin/general-real-qua…

f5a08e2

…nt-refit Signed-off-by: Meng Xin <mxin@nvidia.com> # Conflicts: # tests/unit/algorithms/test_grpo.py

fix: record real quant generation init timing

7f647b9

Signed-off-by: Meng Xin <mxin@nvidia.com>

Merge origin/main into mxin/general-real-quant-refit

5ca6690

Signed-off-by: Meng Xin <mxin@nvidia.com>

docs: clarify real quant export support

a92d14e

Signed-off-by: Meng Xin <mxin@nvidia.com>

github-actions Bot added the Documentation Improvements or additions to documentation label Sep 1, 2026

mxinO changed the title ~~refactor(modelopt): generalize real-quant vLLM refit~~ feat(modelopt): add format-general real-quant vLLM rollout Sep 1, 2026

mxinO added 13 commits August 31, 2026 19:49

fix: validate real quant refit configuration

58c05e2

Signed-off-by: Meng Xin <mxin@nvidia.com>

refactor(modelopt): reuse native real-quant refit lifecycle

cda0480

Signed-off-by: Meng Xin <mxin@nvidia.com>

Add Qwen3 8B mixed real quant example

c047883

Signed-off-by: Meng Xin <mxin@nvidia.com>

docs: clarify distributed quant recipe usage

c7b7c1e

Signed-off-by: Meng Xin <mxin@nvidia.com>

fix(modelopt): enable CUDA graphs for Qwen3 real quant

6db5bf1

Signed-off-by: Meng Xin <mxin@nvidia.com>

fix(modelopt): make refit snapshots Ray serializable

b274971

Signed-off-by: Meng Xin <mxin@nvidia.com>

fix(modelopt): retain IPC weights during layerwise reload

7a06b82

Signed-off-by: Meng Xin <mxin@nvidia.com>

Revert "fix(modelopt): retain IPC weights during layerwise reload"

e4d554d

This reverts commit 7a06b82. Signed-off-by: Meng Xin <mxin@nvidia.com>

fix(modelopt): synchronize collective refit export

48eb2a8

Signed-off-by: Meng Xin <mxin@nvidia.com>

test(modelopt): cover refit export synchronization

d8e125e

Signed-off-by: Meng Xin <mxin@nvidia.com>

Merge latest real-quant PR changes

1e9766b

Signed-off-by: Meng Xin <mxin@nvidia.com>

fix: rendezvous real-quant refit by export group

0d4562e

Signed-off-by: Meng Xin <mxin@nvidia.com>

build: update MBridge for grouped refit export

2ea13a4

Signed-off-by: Meng Xin <mxin@nvidia.com>

mxinO added 6 commits September 9, 2026 08:44

fix(modelopt): exclude quantizer state from refit tasks

e769307

Signed-off-by: Meng Xin <mxin@nvidia.com>

fix(modelopt): make NVFP4 MoE scales reloadable

fa4f711

Signed-off-by: Meng Xin <mxin@nvidia.com>

fix(modelopt): keep real-quant weights reloadable

56ae8f2

Signed-off-by: Meng Xin <mxin@nvidia.com>

Merge remote-tracking branch 'origin/mxin/general-real-quant-refit' i…

ba89207

…nto mxin/refit-export-groups Signed-off-by: Meng Xin <mxin@nvidia.com>

docs: update grouped real-quant status

e603551

Signed-off-by: Meng Xin <mxin@nvidia.com>

docs: record mixed real-quant validation

ebce1a1

Signed-off-by: Meng Xin <mxin@nvidia.com>

mxinO added 2 commits September 9, 2026 20:45

docs: clarify real-quant qualification

b15844d

Signed-off-by: Meng Xin <mxin@nvidia.com>

build: update MBridge documentation dependency

02d35a1

Signed-off-by: Meng Xin <mxin@nvidia.com>

docs: specify grouped MoE validation topology

84ef399

Signed-off-by: Meng Xin <mxin@nvidia.com>

fix: validate real-quant config before setup

46d845c

Signed-off-by: Meng Xin <mxin@nvidia.com>

Merge remote-tracking branch 'origin/main' into mxin/refit-export-groups

79954dd

Signed-off-by: Meng Xin <mxin@nvidia.com>

docs: add real-quant refit flow chart

a883a3e

Signed-off-by: Meng Xin <mxin@nvidia.com>

docs: add real-quant caller path

6b169e8

Signed-off-by: Meng Xin <mxin@nvidia.com>

docs: add caller path PNG

0918406

Signed-off-by: Meng Xin <mxin@nvidia.com>

docs: add future offline export call flow

7158121

Signed-off-by: Meng Xin <mxin@nvidia.com>

mxinO requested review from sharonyu-115 and zhangyimi September 10, 2026 08:25

mxinO added CI:L2 Run doctests, unit tests, functional tests, and convergence tests CI:L1 Run doctests, unit tests, and functional tests and removed CI:L2 Run doctests, unit tests, functional tests, and convergence tests labels Sep 10, 2026

docs: keep architecture figures out of tree

d224c74

Signed-off-by: Meng Xin <mxin@nvidia.com>

Navigation Menu

Uh oh!

feat(modelopt): add format-general real-quant vLLM rollout - #3945

feat(modelopt): add format-general real-quant vLLM rollout#3945
mxinO wants to merge 67 commits into
mainfrom
mxin/general-real-quant-refit

mxinO commented Sep 1, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Sep 1, 2026

Uh oh!

github-actions Bot commented Sep 10, 2026

Uh oh!

github-actions Bot commented Sep 10, 2026

Uh oh!

github-actions Bot commented Sep 10, 2026

Uh oh!

github-actions Bot commented Sep 10, 2026

Uh oh!

github-actions Bot commented Sep 10, 2026

Uh oh!

github-actions Bot commented Sep 10, 2026

Uh oh!

github-actions Bot commented Sep 10, 2026

Uh oh!

github-actions Bot commented Sep 10, 2026

Uh oh!

github-actions Bot commented Sep 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

mxinO commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Dependencies

Online refit call flow

Validation

Qwen3-8B mixed-format qualification

MXFP8 attention projection speed

Uh oh!

copy-pr-bot Bot commented Sep 1, 2026

Uh oh!

github-actions Bot commented Sep 10, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

github-actions Bot commented Sep 10, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

github-actions Bot commented Sep 10, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

github-actions Bot commented Sep 10, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

github-actions Bot commented Sep 10, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

github-actions Bot commented Sep 10, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

github-actions Bot commented Sep 10, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

github-actions Bot commented Sep 10, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

github-actions Bot commented Sep 10, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

mxinO commented Sep 1, 2026 •

edited

Loading