build: bump vllm to 0.29.0

[yfw]

What does this PR do ?

Bumps vLLM from 0.26.0 to 0.29.0 (this branch started from 0.25.1; main moved to 0.26.0 with #3566 on Sept 19). The vLLM 0.29.0 wheels are built against torch 2.13, so the vLLM / Megatron / Automodel / FSDP venvs move with it: torch 2.11.0 → 2.13.0, torchvision 0.26.0 → 0.28.0, triton 3.6.0 → 3.7.1, flashinfer 0.6.14 → 0.6.18, nvidia-cutlass-dsl 4.6.0 → 4.6.2, tilelang 0.1.9 → 0.1.12 in the vLLM venv, llguidance 1.8.0 → 1.7.6 (vLLM 0.29's own <1.8 bound), numpy 2.5.3 → 2.4.6. The sglang and TRT-LLM venvs stay on torch 2.11.0, because their kernels are built against it; torch is now pinned per backend extra and a small IPC-handle shim lets the 2.11 inference venvs open weight handles produced by the 2.13 trainer.

Full nightly suites on H100 (cw_dfw) and GB200 (oci_hsg) were compared recipe by recipe against main's own scheduled nightlies at this PR's merge point. Every regression the bump introduced is fixed on this branch and validated on hardware. The remaining failures are pre-existing on main, infrastructure, or goldens sitting at their bounds, each with the main control that shows it.

Changes

Bullets are prefixed with the commit that made the change.

Dependencies (pyproject.toml / uv.lock)

• 83fa485 / e179799 — vLLM 0.26.0 → 0.29.0 from PyPI (default cu130 wheels for both arches), torch 2.11.0 → 2.13.0, torchvision 0.26.0 → 0.28.0, flashinfer-python/-cubin/-jit-cache 0.6.14 → 0.6.18 (cu130 wheels for both arches, matching the ABI of the 0.29.0 vLLM wheels), nvidia-cutlass-dsl[cu13] 4.6.0 → 4.6.2 in the vllm extra (and 4.5.2 → 4.6.2 in mcore), tilelang <0.1.13 (0.1.9 → 0.1.12 in the vLLM venv), llguidance>=1.7.0,<1.8.0 (1.8.0 → 1.7.6: vLLM's own bound, which main's >=1.3.0 floor bypassed). flash-attn switches from the pinned cu13torch2.10 GitHub wheels to flash-attn==2.8.1 (sdist) because no 2.8.x wheel exists for torch ≥ 2.11.
• e179799 — instanttensor kept to x86_64 via an override. vLLM 0.29 lists instanttensor>=0.1.9 unconditionally; it is the optional load_format="instanttensor" safetensors loader, imported lazily, and NeMo-RL never selects it. 0.2.0 ships aarch64 wheels, so this is a choice to leave an unused dependency out of the GB200 image, not a wheel-availability limit.
• 2e0bdb8 — lockfile regenerated for the Docker uv version.
• 9d8f808 — cutlass-dsl 4.6.2 in every fork that can take it. main already removed the nvidia-cutlass-dsl-libs-base exclusion in feat: add end-to-end TE NVFP4 training with per-token vLLM rollout #3566 (4.6 split the libs-base / libs-cu13 wheels cleanly; the 4.5.x pair overwrote 180 shared files). This PR moves the vllm extra 4.6.0 → 4.6.2 (flashinfer 0.6.18's requirement) and the mcore pin 4.5.2 → 4.6.2 so the training venv does not inherit the overlapping 4.5.x pair, drops feat: add end-to-end TE NVFP4 training with per-token vLLM rollout #3566's quack-kernels==0.6.1 pin (0.29 pins 0.6.4), and keeps a NOTE in pyproject.toml on why the exclusion must not come back. The trtllm fork stays on 4.5.0 via tensorrt-llm.
• 9d8f808 — flash-attn source build bounded with MAX_JOBS=4 via [tool.uv.extra-build-variables]. flash-attn's setup.py sizes its parallelism from the host's free memory (not the cgroup) and cpu_count()//2, with --threads 4 per nvcc; every nemo-ci build attempt on both architectures died with ResourceExhausted after dozens of Killed "$CICC_PATH/cicc". 4×4 builds cleanly on aarch64 (2h47m) and x86 (~2h, alongside TE and flash-mla).
• 9d8f808 — numpy override capped at <2.5 (resolves 2.4.6). vLLM 0.29 pins numba==0.65.0, whose metadata requires numpy<2.5; the numpy>=2.1.0 override forced 2.5.1 past it and numba refused to import in the vLLM venv ("Numba needs NumPy 2.4 or less"). Not fatal by itself — vLLM's optional-module probe catches it — but it left a broken numba in the venv and polluted every trace.
• e991336 — ModelOpt stays at main's 43fd41a58, now with a do-not-move note. The pin must not move past 2ded5aba3 (ray.sub requires submission from NeMo RL home directory, blocking external workflow organization #1550): that commit introduces GroupedQuantizer, which breaks the Megatron fakequant DDP hooks and real-quant dist-ckpt loading (observed on ModelOpt main, 613e5e8b). 43fd41a58 is a release cherry-pick that carries Direct support for megatron bridge checkpoints #2054 (the RoutedExperts vLLM plugin this bump needs; the pre-Direct support for megatron bridge checkpoints #2054 plugin registered on fused_moe.layer.FusedMoE, which 0.29 no longer has) and excludes ray.sub requires submission from NeMo RL home directory, blocking external workflow organization #1550.
• 5200efa — flash-linear-attention / fla-core relocked 0.5.1 → 0.5.2, the version main's lock resolves. fla-core 0.5.1 enabled flash-linear-attention's TileLang kernels wherever tilelang imports (the mcore extra installs it for the fused DSA kernels), including Blackwell, and its TileLang gated-delta-rule backward faults there with CUDA error: misaligned address; 0.5.2 ([Bug] GDN precision error in triton3.5 and h20 fla-org/flash-linear-attention#640) turns TileLang on by default only on Hopper with Triton ≥ 3.4 and keeps Triton elsewhere. The GB200 grpo-qwen3.5-35ba3b-6n4g-async-1off-bf16-trtllm recipe died at step 1 of MegatronPolicyWorker.train() on every run of this branch while main (0.5.2) passes it; it had been mis-cleared against a Sept 10 main control that still carried 0.5.1. This branch's lock had lagged because each main merge kept our uv.lock and re-locked without upgrading transitive packages.
• be440de — every transitive package relocked to at least main's resolution (uv lock --upgrade, with Ray held at main's 2.56.1). About 150 packages had resolved older here than on main for the reason above (pandas 2.3 vs 3.0, starlette, xgrammar, tensordict, nccl4py, cuda-pathfinder, …); 103 move, none of the pinned ones (vLLM, torch per extra, triton, flashinfer, cutlass-dsl, TE, ModelOpt, sglang, TRT-LLM, DeepEP, nixl) change, and the only packages still below main are the deliberate numpy <2.5, llguidance <1.8 and the flash-attn sdist. The nightlies in the Test plan ran on this lock.
• ab6b3f2 — the trtllm venv stays on cuda-bindings / cuda-python 13.3. The relock above moved them to 13.4.x, and cuda-bindings 13.4 removed the reserved field of cudaIpcMemHandle_t that tensorrt_llm 1.3.0rc21's _ipc_utils.py reads while allocating the TP>1 all-reduce fusion workspace; the GB200 grpo-qwen3-1.7b-2n4g-fsdp2-trtllm recipe died at engine start with AttributeError: … has no attribute 'reserved' (450733567) while the TP=1 TRT-LLM recipes passed. The trtllm extra pins both <13.4 (main's versions); the torch-2.13 forks keep 13.4.x, which the vLLM and Megatron recipes passed with.

vLLM source patches / worker (nemo_rl/models/generation/vllm/)

• ae4ebcf — VllmInternalWorkerExtension.synchronize_device renamed to synchronize_sparse_refit_device. vLLM 0.29 adds WorkerBase.synchronize_device ([Bugfix][DP] Synchronize the device on pause completion vllm-project/vllm#52914) and WorkerBase.init_worker asserts that a worker extension never shadows a Worker attribute, so every vLLM engine died at init with AssertionError: Worker class … already has an attribute synchronize_device, which conflicts with the worker extension class. The extension method only drained peers before the sparse-delta refit removed staged batch files; its RPC caller and test follow the rename. An AST scan of both extension classes against 0.29's Worker/WorkerBase shows no other overlap, and a new vllm-marked unit test mirrors vLLM's collision predicate so the next upstream name clash fails in CI rather than on a GPU job.
• ae6507c — ErrorResponse imported from vllm.entrypoints.serve.engine.protocol. vLLM 0.29 moved openai/engine/protocol.py out of the openai package ([Frontend] Move engine/protocol.py out openai folder vllm-project/vllm#54492); VllmAsyncGenerationWorker.post_init_async imported it from the old path when bringing up the OpenAI-compatible server, so every recipe that serves generation over HTTP (Gym, swe1, mopd) died with ModuleNotFoundError: No module named 'vllm.entrypoints.openai.engine'. A new vllm-marked test walks nemo_rl/models/generation/vllm with ast, collects every unguarded from vllm… import statement in the vLLM, Dynamo and ModelOpt generation trees (109 today, mostly lazy, method-local imports no unit test executes) and resolves them against the installed vLLM.
• 710d278 — check_admission proxied through _AsyncLLMHTTPClient. vLLM 0.29's OpenAIServing._preflight calls engine_client.check_admission(n) before every response ([Core] Add max_num_queued_reqs and max_num_queued_tokens for queue size management vllm-project/vllm#49445, queue admission control). The async worker's engine-client wrapper exposes an explicit member surface by design, so every /v1/chat/completions 500ed with AttributeError: '_AsyncLLMHTTPClient' object has no attribute 'check_admission' and the Gym-path recipes died on AsyncTrajectoryCollector aborting: … batch-worker failure(s) exceeded max_generation_failures. Forwarded like the other status-only members; unit-tested.
• e1c16e8 — Tied-embedding aliases dropped before load_weights. vLLM 0.29's AutoWeightsLoader (Fix weight tying vllm-project/vllm#51665) skips a tied alias such as lm_head.weight and then asserts that its canonical model.embed_tokens.weight was loaded in the same load_weights call. Refit streams weights in transport-sized batches, so the two routinely land in different calls and every tied-embedding model refit from a DTensor/automodel policy died at its first weight update (ppo_qwen2_5_1_5b_gsm8k_1n8g_automodel_noncolocated{,_async}, dapo_gemma4_e2b_it_1n8g_fsdp2_automodel, vlm_grpo_gemma4_e4b_geo3k_1n8g_automodel) with ValueError: 'lm_head.weight' was skipped because it is tied to 'model.embed_tokens.weight' … was not found in the checkpoint. The alias never loads anything, so it is now dropped up front using vLLM's own _get_tied_embedding_params and the model's hf_to_vllm_mapper (so Gemma-style renamed parameters are covered), on both the batched refit and the native reload_weights path; the MTP drafter still sees the unfiltered stream. Unit-tested; a vllm-marked test pins the helper's existence.
• a419679 — _patch_vllm_ray_executor_v2_tcpstore_port recognises the upstream fix. vLLM 0.29 ships [Bugfix] Avoid TCPStore port collision for co-located non-DP Ray engines vllm-project/vllm#53666 and #50969: RayWorkerProc.create_dist_init_method binds the torch.distributed TCPStore itself on a kernel-assigned port and holds the socket (self._dist_init_store = store) until init_process_group reuses it, so the probe/bind window this patch closed on 0.25 is gone and _select_tcpstore_port no longer exists. Without this change the patch found no anchor and logged "Engines spanning nodes may fail with EADDRINUSE" on every worker start. It now logs at info and leaves the file alone; the port-arithmetic unit tests skip when the installed vLLM carries the upstream marker, and a new test pins the no-warn/no-edit behaviour against a 0.29-style source. The reserved VLLM_PORT band still governs the MessageQueue and API-server ports.
• 83fa485 — _patch_vllm_shm_broadcast_bind_retry re-targeted: vLLM 0.29 binds the MessageQueue remote socket to port 0 directly (no probe/bind race any more) but ignores VLLM_PORT; the patch restores reserved-band selection with bind retries so engine sockets stay out of the ephemeral range.
• f6993c8 — mm_device_do_normalize defaults to False. vLLM 0.29 ([Model] Fused mm preprocess normalisation on the Device vllm-project/vllm#50411) calls the HF image processor with do_rescale=False, do_normalize=False and re-applies both on the GPU in the vision tower's dtype. The policy normalizes the same images on the CPU in fp32 through the same processor, and the nightly probability-error checks assume identical inputs on both sides. Verified on CPU with transformers 5.12.1: the fused path is exact in fp32 but differs by up to 0.019 (≈1.7 % of the pixel std) in bf16; with the default flipped the Qwen2.5-VL clevr DTensor recipe is back at main-level metrics (token_mult_prob_error 1.019, gen_kl 0.0008, vs 198 / 0.14 with on-device normalization). Opt back in with policy.generation.vllm_kwargs.mm_device_do_normalize=true.

fp8 generation (quantization/fp8.py)

• 1734158 — make_fp8_moe_kernel no longer takes a layer kwarg (0.25 forwarded it only to the FlashInfer TRTLLM experts). Both call sites updated; the unit test's expected kwargs follow.
• 1734158 — process_weights_after_loading_kv re-mirrored on 0.29's BaseKVCacheMethod.process_weights_after_loading: the attention layer no longer has calculate_kv_scales (dynamic scales are a KV-cache dtype now, kv_cache_uses_per_token_head_scales), the static-scale branch keys off is_quantized_kv_cache, and the host _k_scale_cpu/_v_scale_cpu copies are refreshed on refit. Parameters are still kept so refit can update them.

ModelOpt fakequant / real-quant refit (nemo_rl/modelopt/models/generation/)

• ee35b44 / f3fea21 — per-expert input_quantizer._amax routed around vLLM's expert loader, with checkpoint names mapped through hf_to_vllm_mapper first. The Megatron side exports one input_quantizer._amax per expert projection (…experts.16.up_proj.input_quantizer._amax); the vLLM ModelOpt MoE module owns a single fused quantizer per projection group (w13_input_quantizer / w2_input_quantizer). vLLM 0.25 loaded these through the model-level parameter dict, which NeMo-RL patches to include quantizer buffers; 0.29's AutoWeightsLoader hands every experts.* name to RoutedExperts.load_weights, which rewrites it with the expert mapping and resolves the result with a single getattr — a dotted buffer name cannot resolve that way and every fakequant nvfp4 MoE refit died with AttributeError: Layer … has no parameter 'w13_input_quantizer._amax'. New vllm_quant_moe_amax.route_moe_input_quantizer_amax applies the model's hf_to_vllm_mapper (Nemotron-H: backbone. → model.) and then the same rewrite vLLM does, walks the dotted path on the expert module, fans the values in with max, and hands the remaining weights to vLLM's loader. Unit-tested (mapped and unmapped cases); validated by the GB200 distillation_nano3_30ba3b_4n4g_megatron_qa_nvfp4_modelopt_spec recipe.

Distributed (collectives.py)

• 3558f0a — all_gather_into_tensor → all_gather_single (torch 2.13 API).

Docker (docker/Dockerfile, 3rdparty/TensorRT-LLM-workspace/_backend.py)

• 40f5bcc / fe6945a — TRT-LLM wheel mirror: skip the copy when it is already there. main's fix(docker): persist the TRT-LLM wheel #4153 mirrors the content-addressed wheel into /opt/trtllm_wheels in the hermetic layer and has the release stage read it from there. This PR's residual is 3rdparty/TensorRT-LLM-workspace/_backend.py not re-copying a mirror that already exists (a stage inheriting the mirror from an earlier layer no longer duplicates it into its own) and the matching Dockerfile comment.

Audio deps (tools/install_audio_deps.sh, docs/guides/grpo-audio-visual.md)

• 2c0ad3c / 72b3b35 — torchcodec==0.11.1 → 0.16.0 in the test-time audio installer. Per the torchcodec compatibility table, 0.11 is built for torch 2.11 only while 0.12+ use the stable ABI for torch ≥ 2.11; the image now ships torch 2.13. 0.16.0 is what Gym's docker/install_codec_deps.sh already installs, so the two venvs agree. torchaudio==2.11.0 stays: it is the final torchaudio release and carries no torch pin. Guide updated to state the exact pins.

Tests (tests/test_suites/)

• 3e3dea6 — grpo-qwen3-30ba3b-4n8g-megatron-qa-nvfp4.sh accepts ModelOpt's Inserted 723 quantizers line as well as 723 TensorQuantizers found in model. The latter comes from mtq.print_quant_summary, which the Megatron quant worker only calls when it quantizes from scratch; when the cached quantized checkpoint (<model>_modelopt_<cfg-hash>, keyed on quant config + calibration settings, not on the ModelOpt version) exists it restores it and only the Inserted … line is printed.

torch per backend extra (pyproject.toml / uv.lock)

• 237a52c / 04b9dc9 — torch is pinned per extra, not globally. The base dependency is a range, torch>=2.11,<2.14 with torchvision>=0.26,<0.29; the vllm, mcore, automodel and fsdp extras pin torch==2.13.0 / torchvision==0.28.0, and the sglang and trtllm extras pin torch==2.11.0 / torchvision==0.26.0. The extras are declared mutually conflicting under [tool.uv], so uv resolves each one in its own fork and the lockfile carries both torch versions. The driver venv (no extra) resolves 2.13.0 today; the <2.14 ceiling keeps a future relock from silently moving it. This is what makes sglang-kernel 0.4.5 and TRT-LLM usable again: both ship extensions built against torch 2.11 and failed at import on 2.13 (undefined symbol _ZNK2at10TensorBase14const_data_ptr… / _ZNR5torch7Library4_def…). Verified with uv export --frozen --extra <name> per fork and by the sglang / TRT-LLM recipes on both SKUs.
• 94dd20b — CUDA IPC handles from torch 2.13 open in torch 2.11 inference venvs. With the trainer on 2.13 and sglang / TRT-LLM on 2.11, every colocated refit died at the first weight with received sharable handle from a future version of torch that this version does not know how to handle. torch 2.13 ([CUDA] Fix CUDA IPC deserialization mismatch with expandable_segments on FABRIC_HANDLE pytorch/pytorch#179618) bumped the SHAREABLE_HANDLE_VERSION byte in CUDACachingAllocator::shareIpcHandle from 2 to 3 for the expandable-segment header; the 'c' (plain cudaMalloc) payload is byte-identical, and the consumer only checks version <= own. New nemo_rl/utils/cuda_ipc.py::normalize_cuda_ipc_handle rewrites the version byte of 'c' handles to 2 and leaves 'e' handles (whose format did change) and raw 64-byte handles alone; applied in rebuild_cuda_tensor_from_ipc and in the sglang _rebuild_cuda_tensor_modified patch. The shim never consults the consumer: it lowers the version byte of any 'c' handle above 2 and leaves 'e' handles alone (see Notes). 13 unit tests.
• 10e1a86 — DeepSeek V4 refit: model-level post-load hook deferred to the end of the layerwise refit. vLLM 0.29's DeepseekV4ForCausalLM.load_weights now ends with self.process_weights_after_loading(), which recomputes the first layer's hyper-connection broadcast from hc_attn_fn; NeMo-RL streams a refit through buffer-sized load_weights calls while vLLM's layerwise reload still holds that parameter on the meta device, so the 16-node DeepSeek V4 Flash recipe died at its first refit with Cannot copy out of meta tensor. deepseek_v4_fp8.prepare_refit now shadows the hook on the instance, finalize_refit lifts it and runs it once after finalize_layerwise_reload, and restore_refit lifts it on failure too. Unit-tested; see Known issues for the follow-on OOM this uncovered.
• 28beca5 — DeepSeek V4 refit: two more 0.29 changes handled. (a) 0.29 gives DeepseekV4ForCausalLM a packed_modules_mapping in which wkv is a shard of two fused modules (fused_wqa_wkv and fused_wkv_wgate); the fp8 module lookup resolved shards by leaf name alone, sent attn.wkv to a module that does not exist, and refit the weight unquantized. Shard names listed under more than one fused module are now left to the parent-aware DeepSeek V4 remap. (b) 0.29 split the layerwise-reload skip list into SKIP_TENSORS (stay off the meta device) and SKIP_LOAD_TENSORS (loader not wrapped or counted); 0.25.1 did both with one set. The immediately loaded expert tensors were therefore counted, vLLM ran process_weights_after_loading on each expert layer as soon as its last expert arrived, and finalize_refit converted the already kernel-layout weights and scales a second time (verified with a probe: scales doubled twice per refit). prepare_refit registers the names in both sets and returns a per-set SkipNames record. Both were caught by running the vllm-marked unit tests that main added with feat: add DeepSeek V4 Flash GRPO support - Automodel path #3595/fix(vllm): support BF16 TRTLLM NCCL reshard refit #3659 against 0.29; those tests are ported to 0.29's API in the same commit (WeightsMapper.get_rename_mapper, Fp8MoEMethod reading the MoE config, the import-target test skipping try/except ImportError fallbacks).
• 31dc3a8 — torch 2.13 removed the named-tensor API. main's Mooncake TQ checkpoint adapter (feat(data-plane): add Mooncake storage checkpoints #3898) guarded its fast path with produced.names == (None, None); Tensor.names no longer exists on 2.13, so _physical_keys raised AttributeError (caught by test_tq_mooncake_checkpoint.py). The guard now treats a missing attribute as "no named dimensions". The same commit marks the tied-alias unit tests vllm and adds vLLM 0.29's ErrorResponse module to the chat-template wiring test's fake vLLM tree, so the L0 shards' no-vLLM pass over the vLLM test files is green again.
• main merged through a9c3178 (Sept 22). Notable merges: chore(deps): bump mooncake and wandb #4179 (mooncake 0.3.13.post1 / wandb 0.30.0), build: exclude PyAV from shipped container #4184 (PyAV exclusion) and feat: add end-to-end TE NVFP4 training with per-token vLLM rollout #3566 (TE NVFP4 end-to-end; on main it bumped vLLM to 0.26.0 / flashinfer 0.6.14 / cutlass-dsl 4.6.0, moved TE to the 2.18.0 PyPI wheels and ModelOpt to 43fd41a58, and refactored the fp8 module lookup into quantization/utils.py). Resolution: this branch's vLLM 0.29.0 / flashinfer 0.6.18 / cutlass-dsl 4.6.2 pins stay (0.29 pins quack-kernels==0.6.4, so feat: add end-to-end TE NVFP4 training with per-token vLLM rollout #3566's 0.6.1 pin is not taken); TE 2.18.0 and ModelOpt 43fd41a58 (a release cherry-pick that carries Direct support for megatron bridge checkpoints #2054 and still excludes ray.sub requires submission from NeMo RL home directory, blocking external workflow organization #1550's GroupedQuantizer) come from main; the ambiguous-shard fix from 28beca5 is ported into the new resolver. feat: add fp32 LM head toggle #4096's nemotron_h.py fp32 LM head patch anchors occur exactly once in 0.29's copy and its anchor tests pass. The Sept 21–22 merges (fix(ppo): rebuild the critic optimizer on a warm start #4213, fix(vllm): strip staged token arrays from capture responses #4206, feat(mopd): support multiple teacher checkpoints in full-vocabulary o… #4045, fix(LOO-Norm-Advantage): Fix spike in LOO normed advantage. #4158, feat(nemo-gym): wire sharded stacks into training #3373, fix(nemo-gym): preserve replica identity during routing #3374, ci: install research folder dependencies & discover research test suites #4216, test: re-enable nemotron omni clevr megatron single-controller L1 test #4231, feat(draft): preserve split-step normalization parity #3706) had no conflicts.

torch 2.13 / vLLM 0.29 fallout found by the GitHub CICD and the rerun nightly (0cc6fab, 8e97d91)

• 0cc6fab — Async checkpoint daemons join the training TCPStore instead of binding a port. torch 2.13's process-based DCP checkpointer (_async_process_executor) rendezvous its save daemons on a port rank 0 probes with get_free_port(); the port was re-taken before the daemon bound it and every DTensor async save died at the first checkpoint with DistNetworkError … EADDRINUSE (the L1 GRPO resume test, the converter round-trip test, and the llama3.2-1b fsdp2tp2 tq_mooncake nightly recipe at step 10). Surfaced by Automodel 72daceffa making daemon creation synchronous across ranks; main on torch 2.11 does not reach it. init_checkpointer now sets torch's DCP_USE_PREFIX_STORE=1 when MASTER_ADDR/MASTER_PORT are present, so the daemons attach to the process group's store under a prefix. Three unit tests.
• 0cc6fab — nvfp4_pertoken.py (from feat: add end-to-end TE NVFP4 training with per-token vLLM rollout #3566) no longer passes layer= to make_nvfp4_moe_kernel. vLLM 0.29 dropped the parameter (it takes per_token_activation); the new GB200 te_nvfp4_pertoken_quick recipe died with TypeError: unexpected keyword argument 'layer'; a unit test binds the rebuild's kwargs to the installed factory's signature.
• 0cc6fab / 8d1f98a — 2.long_generation_decode_vs_prefill.py gains --gpu-memory-utilization; the L1 test passes 0.7 on GB200 (at 0.8 the FlashInfer TRT-LLM BF16 MoE workspace no longer fit next to 0.29's KV cache) and keeps 0.8 on 80 GiB parts, where 0.7 leaves no KV cache at all.
• 8e97d91 — test_vllm_http_server ignores the usage.completion_tokens_details field vLLM 0.29 added to UsageInfo.
• af7d9e0 — Routed-experts capture for the per-token NVFP4 MoE goes through the router on 0.29. Router replay worked on 0.26 because NeMo-RL's _patch_vllm_moe_routed_experts_capture fires router.select_experts on the monolithic fused-MoE branch when the router carries a capture callback. 0.29's bind_routed_experts_capturer binds monolithic kernels to the experts object instead (the FlashInfer launch then fills routing_replay_out itself) and rejects kernels without in-kernel capture; the per-token method rebuilds its kernel on every refit, so that binding would be lost after the first weight update and replay would silently see all-zero routes. host_captured_experts_cls wraps the method's experts class to report no in-kernel capture, and _patch_vllm_routed_experts_capture_router_fallback makes the 0.29 binder fall back to router.set_capture_fn for such kernels instead of raising. Kernels that support in-kernel capture keep using it. 7 unit tests (anchor against the installed vLLM, idempotency, fail-closed, an executable check of the patched binder, the wrapper).
• a204116 — review round 1: the test_init_checkpointer_* tests carry the automodel mark so the Automodel shard runs them; a per-token test binds the kernel rebuild to the installed make_nvfp4_moe_kernel signature; the import-target walk covers the Dynamo and ModelOpt generation trees; the DeepSeek V4 tests pin vLLM's two distinct skip sets; normalize_cuda_ipc_handle is annotated; stale comments (RayWorkerProc, TRT-LLM mirror, instanttensor, fp8 docstring) and two docs mentions of vLLM 0.25.1 are corrected.
• Also in the diff: docs/guides/dynamo-generation.md and docs/guides/grpo-audio-visual.md version / pin one-liners, and pyrefly.toml includes for vllm_quant_moe_amax.py and cuda_ipc.py.

Known issues not fixed here

Issue	Recipes	Status / proposed fix
grpo-deepseek-v4-flash-0731-16n8g-automodel-cp8ep128 (H100) — fails on the trainer side, as on main	1	With the 0.29 refit fixes (10e1a86, 28beca5) the refit and step-1 generation complete and the run dies in DTensorPolicyWorkerV2.get_logprobs() with DeepEP error: timeout (dispatch CPU) — the same point and error as the main control at this PR's earlier base (446267140). main's Sept 20 / 21 nightlies fail it earlier, at vLLM engine start. vLLM path at parity; trainer side broken on main.
grpo-qwen3-30ba3b-4n4g-megatron-te-nvfp4-pertoken-quick (GB200) — resolved; kept here as the record of a 0.29 behaviour change	1	Recipe landed on Sept 19 (#3566); three 0.29 changes hit it. (1) make_nvfp4_moe_kernel(layer=) was dropped in 0.29 — fixed in 0cc6fab. (2) 0.29 binds routed-experts capture for monolithic kernels to the kernel's experts object and lets the FlashInfer launch fill routing_replay_out itself; this method rebuilds its kernel on every refit, so router replay would have silently lost its routes after the first refit. af7d9e0 wraps the experts class to report no in-kernel capture and adds a source patch so 0.29's binder falls back to the router hook (the path NeMo-RL's existing _patch_vllm_moe_routed_experts_capture fires, and how this recipe captured on 0.26) instead of raising. (3) With (1) in place the recipe died in the first post-refit generation with CUDA error: an illegal memory access in the FlashInfer TRT-LLM NVFP4 MoE kernel (450220649). That launch had the in-kernel replay buffer attached — the one FlashInfer launch this recipe exercises that 0.26 never ran. With af7d9e0 routing the capture through the router (so the kernel launches without a replay buffer) the recipe passes 20/20 (450730474) with gen-KL 0.0083 and median prob-error 1.060, matching main's 0.0082 / 1.058, and router-replay validation on. The per-token FlashInfer launch with routing_replay_out attached remains unusable on flashinfer 0.6.18 and is worth reporting upstream. main passes the recipe on Sept 20 and aborts on Sept 21 in the HybridEP handle import (see W4A4 below).
Colocated Megatron + sglang dies at the step-10 async checkpoint — AttributeError: module 'torch.multiprocessing.reductions' has no attribute '_rebuild_cuda_tensor_original'	2 (H100 2n8g and GB200 1n4g megatrontp1_sglang)	Pre-existing on main (Sept 20 and 21 fail identically); refits 1–10 through the new torch-2.13→2.11 IPC path work. The colocated fsdp2 + sglang recipe passes on H100 (450/450) and on GB200 reaches step ~295/450 before its 150-min Slurm limit, which main also hits. Non-colocated sglang passes on both SKUs.
NVFP4 W4A4 real-quant on GB200 — grpo-qwen3-30ba3b-4n4g-megatron-qa-nvfp4-w4a4-real	1	Two failure modes, both seen on main too. (a) MegatronQuantPolicyWorker ranks abort at step 1 in DeepEP's HybridEP buffer allocation (cuMemImportFromShareableHandle in NVLCoordinator::open_handles_from_other_ranks, then SIGABRT) — main Sept 21, 19 and 18 abort identically, Sept 20 and 16 pass; node/fabric dependent. (b) When it runs, it trains 2/2 with reward and accuracy passing and js_divergence_error[2] at the 0.007 bound (0.00694 / 0.00718 on the two 0.29 samples vs 0.0039–0.0044 on main; gen-KL 0.023 vs 0.015). The H100 W4A16 recipe matches main exactly, so (b) is the NVFP4 activation-quant kernels on Blackwell, not NVFP4 weights generally.
nixl-cu13 1.3.0 nixl_ep extension built against torch 2.11 — nixl_ep_cpp.cpython-313-…so: undefined symbol _ZN3c104impl3cow23materialize_cow_storage…	none	Non-fatal: vLLM's has_nixl_ep() probe catches it and disables the optional nixl_ep all2all backend, but the traceback appears in every vLLM worker log and nemo-ci's error extractor picks it as the "cause" of unrelated failures. Bump nixl to a torch-2.13 build in a follow-up.
grpo-nemotron3-super-120BA12B-8n4g-megatron (GB200) — host OOM during the step-5 checkpoint save	1	On main this recipe dies at its first refit (Unquantized FlashInfer TRTLLM refit does not yet support a co-trained MTP drafter; Sept 20 and 21). On 0.29 it refits and trains all 5 steps, then the Megatron dist-ckpt save host-OOMs the 4-GPU GB200 nodes. The save had never been reached before; not a vLLM path.
mopd-qwen3-1.7b-3n4g-megatron-pack-single-controller-fullvocab (GB200) — never starts: pyxis cannot extract the sandbox squashfs	1	Only USES_SANDBOX=1 recipe in the GB200 nightly; main Sept 20 and 21 fail identically at Slurm step 0. nemo-ci provisioning on oci_hsg. The H100 3n8g twin passes.
Goldens at their bounds	4	llama3_2_1b_instruct_1n4g_fsdp2tp1_v3 (GB200) step time 14.8–15.1 s vs < 14.5 (main 13.7–15.1 s). clevr_8n4g_megatron_generation_v1 (GB200, Megatron generation) final accuracy 0.59–0.62 against > 0.6 / +0.01 (main 0.59–0.64). mxfp8_rollouts_flashinfer (GB200, Megatron generation): 10/10, one-step prob-error spike and 57 s steps vs < 48 (main fails the same recipe on prob-error and step time). sft_gpt_oss_20b_1n8g grad_norm[50] > 10.0: PR 9.74–10.12, main 9.83–10.20.
Gym server spin-up flake	3	A Gym server process exits during spin-up with no output and the NemoGym actor loses the GCS; main Sept 20 hit it on both mopd recipes. All three passed on retry.
New 0.29 default: FlashInfer TP all-reduce (VLLM_ALLREDUCE_USE_FLASHINFER=1)	TP > 1 recipes	No regression seen; opt out via vllm_cfg.env_vars if one appears.

Notes for reviewers

• Two torch versions in one lockfile. Each extra pins its own torch inside a uv conflict fork; uv sync --extra sglang and uv sync --extra trtllm produce torch-2.11 venvs, everything else torch 2.13. The base range torch>=2.11,<2.14 is what the driver venv resolves from; the ceiling is deliberate so a relock cannot move the driver past what the inference venvs can talk to. If a future sglang-miles / TRT-LLM release moves to torch 2.13, drop their pins; the shim then sees only version-3 handles on both sides and can be removed.
• The IPC shim only touches 'c' handles, and never consults the consumer. It lowers the version byte of any 'c' handle above 2. torch 2.13 changed the wire format of expandable-segment ('e') handles along with the version byte; those are left alone. The trainer emits 'e' handles under PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, which vLLM (same torch) opens fine and a torch 2.11 consumer (sglang / TRT-LLM, colocated) rejects with torch's generic "future version" error, so keep that allocator setting off for those two backends; none of the shipped sglang / TRT-LLM recipes sets it.
• Lock parity with main. After be440de the lock is at or above main's resolution for every package except the deliberate pins; the lock diff is therefore larger than the dependency bullets alone would suggest. Compare against main with git show origin/main:uv.lock if in doubt.
• Async checkpoints and DCP_USE_PREFIX_STORE. The flag is torch's own opt-in (torch/distributed/checkpoint/_async_process_executor.py); it is set only when the training rendezvous variables are present, and an explicit value in the environment wins. The daemons then need no free port at all, which is also the right behaviour on hosts where the ephemeral range is contended.
• Gym. docker/Dockerfile still forces Gym's server venvs to NEMO_GYM_VLLM_VERSION=0.25.1, which works against this branch (all Gym-path recipes pass). Gym is being bumped separately in build: bump vllm to 0.29.0 Gym#3505 (vLLM 0.29 + flashinfer 0.6.18; it has to drop Gym's pycountry exclusion because vLLM 0.29 imports mistral_common eagerly); a follow-up RL PR will bump the Gym submodule and that ARG together once it merges.
• The cutlass change removes a [tool.uv] exclusion that was load-bearing at 4.5.x (180 overlapping files, verified from the wheels; 0 at 4.6.x). uv pip install of the overlapping pair silently picks a winner per file, which is why the mcore pin moved to 4.6.2 too.
• instanttensor is the one uv override that drops a dependency on one arch. 0.2.0 does ship aarch64 wheels; the skip keeps an unused loader out of the GB200 image, and lifting the marker is all it takes to enable load_format="instanttensor" there.
• mm_device_do_normalize=False flips a vLLM default that, in 0.29, only Qwen2-VL / Qwen2.5-VL opt into (every other VLM is forced off by vLLM itself), so it affects those two families. It restores the 0.25 behaviour byte-for-byte on the processor side; the cost is the CPU normalize vLLM's change was avoiding.
• ModelOpt stays at main's 43fd41a58; this PR adds the "do not move past ray.sub requires submission from NeMo RL home directory, blocking external workflow organization #1550" note in pyproject.toml (GroupedQuantizer, see the e991336 bullet).
• The vllm-marked unit suite (429 items under tests/unit/models/generation, utils, policy on this head; 2 need a CUDA device) was re-run against the branch lock + vLLM 0.29 after every main merge, alongside the no-vLLM pass the L0 shards do over the same files.

Test plan

Nightlies. Full nightly suites on both SKUs on the final lock (be440de; the later commits touch a functional-test script, the per-token NVFP4 capture path, tests, comments, docs, and — in ab6b3f2 — the trtllm venv's cuda-bindings pin, validated separately below). H100 lane 69151691 and GB200 lane 69183689, both complete. Both are compared recipe by recipe with main's Sept 20 scheduled nightly (68861701, RL 880a37a — exactly the merge point of d104085, vLLM 0.26.0) and Sept 21 (69003177, RL 612d527, one commit behind the 094a932 that 51d7ee4 merged). Two earlier full passes drove the fixes: fe6945a (H100 68529938, GB200 68535477) found the DeepSeek V4 refit changes; 8e97d91 (H100 68945599, GB200 68937624) found the three torch-2.13 items in 0cc6fab and the fla-core lag in 5200efa.

Targeted validation of the fixes made after 8e97d91. EADDRINUSE fix, H100 69074478: llama3_2_1b_instruct_1n8g_fsdp2tp2_temp0_8_topp0_9_topk50_tq_mooncake saves every checkpoint and runs to step 470/500 before the recipe's own 180-min Slurm limit, exactly where main stops. fla-core relock, GB200 69076114: qwen3_5_35ba3b_6n4g_async_1off_bf16_trtllm trains 16 steps with gen-KL 0.0010–0.0012 and no fault, where every earlier 0.29 run died at step 1, then hits Disk quota exceeded on the oci_hsg CI filesystem at step 17 (450537401); w4a4_real aborted a fourth time in the DeepEP handle import that main hits too. trtllm cuda-bindings pin, GB200 69220858: qwen3_1_7b_2n4g_fsdp2_trtllm on ab6b3f2 passes 10/10 (gen-KL 0.0010, reward 0.875, 22 s/step; 451393519). Per-token NVFP4 capture path, GB200 69135677: te_nvfp4_pertoken_quick passes 20/20 with main-level goldens. clevr_8n4g_megatron_generation_v1 rerun (GB200 69074482): 50/50 steps, lands in the same 0.59–0.64 final-accuracy band as main.

SKU	Recipes	Pass	Fail	Running	PR passes, main fails	PR fails, main passes
H100 (cw_dfw)	134	109	25	0	1	5
GB200 (oci_hsg)	47	32	15	0	1	6

The "PR fails, main passes" rows are itemised in the table below with their cause. Several GB200 jobs on this lane spent an hour or more in the code-snapshot copy while the oci_hsg CI project's Lustre quota was full; the ones that then died on Disk quota exceeded are marked as infrastructure.

GitHub CICD (L0 unit shards incl. the no-vLLM and --vllm-only passes over the vLLM tests, L1 functional tests on both SKUs) is green on the current head ab6b3f2 (run 35711529567), as it was on af7d9e0. One H100 shard, L1_Functional_Tests_Megatron_4, is flaky on this branch and on main's runners: in two of five runs grpo_megatron_generation_colocated.sh trains its 2 steps and the driver then exits non-zero without output; a plain re-run passes, and the GB200 copy of the shard passes every time.

Results — nightly, final lock be440de (both lanes complete)

PR test pipelines: H100 69151691 · GB200 69183689. main controls: the Sept 21 scheduled nightly 69003177 (RL 612d527, one commit behind the 094a932 that 51d7ee4 merged) and the Sept 20 one 68861701 (RL 880a37a, exactly the merge point of d104085; vLLM 0.26.0 on both). One row per recipe (checkout bounces retried; latest attempt shown).

• H100: 109 pass · 25 fail · 0 running
• GB200: 32 pass · 15 fail · 0 running

PR fails, main passes (11)

Test	PR	main Sept 21	main Sept 20	Notes
llm_grpo_llama3_2_1b_instruct_1n4g_fsdp2tp1_v3 GB200	❌ failed 165 min	✅	✅	Marginal step-time golden. 500/500 steps, correctness checks pass; mean(total_step_time, -6, -1) < 14.5 misses at 14.9–15.1 s on every PR lane (14.92 on 8e97d91, 15.07 on be440de). PR runs 14.8–15.1 s; main 13.9 s on Sept 20 (pass), 13.7–15.1 s over Sept 8–9. About 5 % on a 1-GPU-per-node recipe whose generation share is unchanged; see Known issues.
llm_grpo_nanov3_30BA3B_4n4g_megatron_generation_noncolocated_mxfp8_rollouts GB200	❌ failed 93 min	✅	❌	Single-step prob-error outlier (Megatron generation, MXFP8 rollouts). 10/10 steps; gen-KL 0.028–0.043 on every step, median token_mult_prob_error 1.25 and step time 24.8 s all pass, but token_mult_prob_error[10] < 2.0 reads 464.6 — one outlier token on the last step while the distribution-level check stays flat. Same shape as the _flashinfer twin's step-10 spike (22.8) on the previous lane and main's Sept 20 miss on that twin (median 1.56). Passed on the 8e97d91 lane and on main Sept 21; vLLM is not in this recipe's loop.
llm_grpo_qwen2_5_math_1_5b_instruct_1n4g_megatron_single_controller_sync GB200	❌ failed 377 min	❌	✅	Infra: oci_hsg CI Lustre quota. The job sat in the code-snapshot copy while the project quota was full, then died on Disk quota exceeded before its first step. Passed on the 8e97d91 lane (single-controller sync, 450/450) and on main Sept 20; main Sept 21 failed the same recipe without a metric table. (main Sept 21: failed.)
llm_grpo_qwen3_1_7b_2n4g_fsdp2_trtllm GB200	❌ failed 251 min	✅	✅	Regression from the transitive lock upgrade (be440de), fixed in the follow-up commit. AttributeError: 'cuda.bindings.runtime.cudaIpcMemHandle_t' object has no attribute 'reserved' in tensorrt_llm/_ipc_utils.py::open_ipc_memory while allocating the TP>1 all-reduce fusion workspace at engine start. uv lock --upgrade had moved cuda-bindings / cuda-python 13.3.1 → 13.4.x (main is on 13.3.1), and 13.4 dropped the reserved field TensorRT-LLM 1.3.0rc21 reads. The trtllm extra now pins both <13.4; on that pin the recipe passes 10/10 (451393519, gen-KL 0.0010). The TP=1 TRT-LLM recipes (qwen2_5_0_5b_1n4g_megatron_trtllm_noncolocated_async, qwen3_5_35ba3b_6n4g_async_1off_bf16_trtllm) never allocate that workspace and passed on the same lock.
llm_grpo_qwen3_30ba3b_4n4g_megatron_qa_nvfp4_w4a4_real GB200	❌ failed 152 min	❌	✅	Pre-existing on main. MegatronQuantPolicyWorker ranks abort at step 1 in DeepEP's HybridEP buffer allocation: cuMemImportFromShareableHandle fails in NVLCoordinator::open_handles_from_other_ranks and the process SIGABRTs before any golden. main fails identically on Sept 21 (449569744), Sept 19 (447326502) and Sept 18 (445908944); it passed on Sept 20 and 16; our retry (449824224) aborted the same way. Node/fabric-dependent. When the recipe does run on 0.29 (two earlier samples) it trains 2/2 with reward and accuracy passing and js_divergence_error[2] at the 0.007 bound (0.00694 / 0.00718; main 0.0039–0.0044) — see Known issues. (main Sept 21: Signal/OOM-Kill.)
llm_grpo_qwen3_30ba3b_4n4g_megatron_te_nvfp4_pertoken_quick GB200	❌ failed 246 min	❌	✅	Regression (new recipe from #3566); fixed in 0cc6fab + af7d9e0, validated. make_nvfp4_moe_kernel() got an unexpected keyword argument 'layer' (vLLM 0.29 dropped the parameter) is fixed in 0cc6fab. With that fix (450220649) the recipe refits and dies in the first generation with CUDA error: an illegal memory access in the FlashInfer TRT-LLM NVFP4 MoE kernel. 0.29 also changed how routed experts are captured for monolithic kernels (bound to the kernel's experts object, which this method rebuilds on every refit, so router replay would have silently lost its routes after the first refit); af7d9e0 routes the capture through the router hook NeMo-RL already patches in, as on 0.26, so the kernel launches without a replay buffer. With both fixes the recipe passes 20/20 (450730474: gen-KL 0.0083, median prob-error 1.060 — main Sept 20: 0.0082 / 1.058; router replay validation on). main passes on Sept 20 and aborts on Sept 21 in the HybridEP handle import (see W4A4). (main Sept 21: Uncaught Exception.)
llm_grpo_gemma3_1b_it_1n8g_fsdp2tp1_tq_simple H100	❌ failed 141 min	✅	✅	Step-time golden under lane contention. 400/400 steps, median(token_mult_prob_error) < 1.1 passes at 1.009; mean(total_step_time, -6, -1) < 14 misses at 19.3 s while ~120 lane jobs shared the cluster. Passed on the 8e97d91 rerun and on main Sept 20/21 in quieter lanes.
llm_grpo_llama3_2_1b_instruct_1n8g_megatron_temp0_8_topp0_9_topk50_tq_mooncake H100	❌ failed 184 min	❌	✅	Borderline recipe duration (main too). 489/500 steps at the 150-min Slurm limit with healthy refits; the same recipe finished 500/500 on the 8e97d91 lane (187 min wall, incl. queue) and main sits on the same edge (Sept 20 pass at 171 min, Sept 21 TIMEOUT at 176 min). (main Sept 21: Slurm time limit.)
llm_mopd_qwen3_1_7b_3n8g_megatron_pack H100	❌ failed 48 min	✅	❌	Completed; Slurm TIMEOUT at teardown (launch contention). 5/5 steps, both goldens pass, exit 0; 12 min to cluster ready against an 18-min limit. Passed on the 8e97d91 rerun (retry) and on main Sept 21. Not retried for the same reason as the SFT recipe above. (main Sept 20: Signal/OOM-Kill.)
llm_sft_gpt_oss_20b_1n8g_fsdp8ep8_automodel H100	❌ failed 42 min	✅	❌	Marginal golden (SFT, no vLLM). grad_norm[50] > 10.0 misses at 9.88; PR runs 9.74–10.12, main 9.83–10.20 (fails Sept 20 at 9.83, passes Sept 21). The bound sits inside the noise of this recipe on both branches.
llm_sft_qwen3_0_6B_1n8g_megatron_yarn_128k H100	❌ failed 48 min	✅	✅	Completed; Slurm TIMEOUT at teardown (launch contention). 20/20 steps, all three loss goldens pass and the recipe exits 0, but the job took 11 min to reach cluster ready (about 120 lane jobs launched together after the checkout-bounce retries) and the 15-min Slurm limit fired during teardown. SFT, no vLLM. Not retried: a retry of a completed recipe short-circuits on the existing metrics.

PR fails, main had no usable result (2)

Test	PR	main Sept 21	main Sept 20	Notes
llm_dpo_qwen2_5_math7b_1n8g_megatron_fused_linear_logprobs H100	❌ failed 48 min	❌	❌	Pre-existing on main. DPO, no vLLM; train/accuracy[10] >= 0.5 at 0.406. main bounced at checkout on Sept 20 and 21; fails on Sept 8 and 9. (main Sept 21: died at checkout (shared-workspace Bad address); Sept 20: died at checkout (shared-workspace Bad address).)
llm_grpo_nanov3_30ba3b_3n8g_megatron_dynamo_swe1 H100	❌ failed 47 min	❌	❌	Pre-existing on main. ValueError: NeMo Gym does not support generation backend 'dynamo'; main fails whenever it gets past checkout (Sept 8, 9; bounced Sept 20/21). (main Sept 21: died at checkout (shared-workspace Bad address); Sept 20: died at checkout (shared-workspace Bad address).)

PR fails, main fails (27)

Test	PR	main Sept 21	main Sept 20	Notes
llm_dpo_nanov3_30B3AB_1n4g_fsdp4ep4_automodel GB200	❌ failed 254 min	❌	❌	Pre-existing on main. DPO, no vLLM. mean(total_step_time, -5, -1) < 5 at 17.3 s; main Sept 20 / 21 fail the same check at 16.5 / 16.1 s plus loss goldens.
llm_grpo_moonlight_16ba3b_4n4g_megatron GB200	❌ failed 29 min	❌	❌	Pre-existing on main. KeyError: decoder.layers.N.self_attention.linear_q_proj.layer_norm_weight from model not in checkpoint at Megatron load; main Sept 20 and 21 fail identically.
llm_grpo_nanov3_30BA3B_4n4g_megatron_generation_colocated_reshard_async_gym GB200	❌ failed 257 min	❌	❌	Pre-existing on main. Megatron generation; Gym rejects the rollouts (pydantic ValidationError … NeMoGymChatCompletionMessageForTraining) until the time limit. main Sept 20 and 21 fail identically. On the be440de lane the node died first (CUDA driver error: uncorrectable NVLink error).
llm_grpo_nanov3_30BA3B_4n4g_megatron_generation_noncolocated_mxfp8_rollouts_flashinfer GB200	❌ failed 256 min	❌	❌	Pre-existing on main. Now trains 10/10 (main's #3630 fixed the earlier abort) and misses two goldens: token_mult_prob_error[10] < 2.0 (a single-step spike to 22.8; median 1.32 passes, gen-KL 0.042 passes) and median(total_step_time) < 48 (57.2 s; 55.9 s on the be440de lane, where the prob-error checks passed). main Sept 20 fails the same recipe on median(token_mult_prob_error) < 1.5 (1.56) and the same step-time bound (55.1 s); Sept 15–19 fail too. Megatron generation with FlashInfer MXFP8 rollouts; vLLM is not in the loop.
llm_grpo_nanov3_30ba3b_4n4g_megatron_qa_nvfp4_w4a16_real GB200	❌ failed 251 min	❌	❌	Pre-existing on main. Completes 1/1 and misses gen_kl_error[1] < 0.003 (0.0116) and max(token_mult_prob_error) < 1.05 (1.094); main Sept 20 misses the same two checks with the same values (0.0112 / 1.094).
llm_grpo_nemotron3_super_120BA12B_8n4g_megatron GB200	❌ failed 44 min	❌	❌	Changed, not vLLM. On main the recipe dies at its first refit (Unquantized FlashInfer TRTLLM refit does not yet support a co-trained MTP drafter; Sept 20 and 21). On 0.29 it refits and trains all 5 steps, then the Megatron dist-ckpt save host-OOMs the 4-GPU nodes. See Known issues.
llm_grpo_qwen2_5_math_1_5b_instruct_1n4g_fsdp2tp1_sglang GB200	❌ failed 170 min	❌	❌	Pre-existing on main. Colocated sglang on GB200 reaches step ~295/450 and hits the 150-min Slurm limit; main Sept 20 and 21 hit the same limit.
llm_grpo_qwen2_5_math_1_5b_instruct_1n4g_megatrontp1_sglang GB200	❌ failed 366 min	❌	❌	Pre-existing on main. Colocated sglang dies at the step-10 async checkpoint on torch.multiprocessing.reductions._rebuild_cuda_tensor_original; main Sept 20/21 fail identically (same step, same attribute) and then sit until the Slurm limit.
llm_mopd_qwen3_1_7b_3n4g_megatron_pack_single_controller_fullvocab GB200	❌ failed 215 min	❌	❌	Infra (main too). Slurm step 0 fails before Ray starts: pyxis failed to create container filesystem extracting the sandbox squashfs on oci_hsg. main Sept 20 and 21 fail identically; the H100 3n8g twin passes.
llm_dapo_nanov3_5_30BA3B_4n8g_automodel H100	❌ failed 12 min	❌	❌	Pre-existing on main. ValueError: Couldn't instantiate the backend tokenizer (HF asset not in the CI cache); main Sept 20 and 21 fail identically.
llm_dpo_mistral_nemo_instruct_2407_1n8g_fsdp2tp8_actckpt_long H100	❌ failed 76 min	❌	❌	Pre-existing on main. DPO, no vLLM; train/loss[1] < 0.70 at 0.7087 — main Sept 20 fails on the identical value (0.7087241); Sept 21 could not submit to Slurm.
llm_dpo_nanov3_30B3AB_1n8g_fsdp8ep8_automodel_v2 H100	❌ failed 57 min	❌	❌	Pre-existing on main. DPO, no vLLM; mean(total_step_time, -5, -1) < 5 at 5.32 s; main Sept 20 fails the same check (5.20 s) plus two loss goldens; Sept 21 could not submit to Slurm.
llm_grpo_deepseek_v4_flash_0731_16n8g_automodel_cp8ep128 H100	❌ failed 37 min	❌	❌	vLLM-side regressions fixed (10e1a86, 28beca5); still fails on the trainer side like main. The refit and step-1 generation complete; the run dies in DTensorPolicyWorkerV2.get_logprobs() with DeepEP error: timeout (dispatch CPU), the same point and error as the main control at this PR's earlier base (446267140). main's Sept 20 / 21 nightlies (vLLM 0.26.0) fail the recipe earlier, at vLLM engine start (RuntimeError: Worker failed with error '' ×11, then the step is cancelled).
llm_grpo_llama3_2_1b_instruct_1n8g_fsdp2tp2_temp0_8_topp0_9_topk50_tq_mooncake H100	❌ failed 215 min	❌	❌	Regression, fixed in 0cc6fab. Dies at the step-10 async checkpoint: torch 2.13's process-based DCP checkpointer rendezvous the save daemons on a get_free_port() probe that is re-taken before the daemon binds it (DistNetworkError … EADDRINUSE). Same crash in the GitHub L1 GRPO checkpoint test and the converter test on this head. init_checkpointer now opts the daemons into DCP_USE_PREFIX_STORE=1 so they join the training TCPStore instead of binding a port. Validated on the fixed head (450067839: no EADDRINUSE, every 10-step checkpoint saved, 470/500 steps until the recipe's 180-min Slurm limit — exactly where main (torch 2.11, never reaches the bug) stops on Sept 20 and 21). What remains is that pre-existing time limit.
llm_grpo_moonlight_16ba3b_4n8g_megatron H100	❌ failed 17 min	❌	❌	Pre-existing on main. KeyError: decoder.layers.N.self_attention.linear_q_proj.layer_norm_weight from model not in checkpoint at Megatron load; main Sept 20 and 21 fail identically.
llm_grpo_moonlight_16ba3b_4n8g_megatron_fp8_e2e H100	❌ failed 12 min	❌	❌	Pre-existing on main. main Sept 20 and 21 fail this recipe at the same point (Megatron load).
llm_grpo_moonlight_16ba3b_4n8g_megatron_tq_simple H100	❌ failed 38 min	❌	❌	Pre-existing on main. KeyError: decoder.layers.N.self_attention.linear_q_proj.layer_norm_weight from model not in checkpoint at Megatron load; main Sept 20 fails identically, Sept 21 bounced at checkout.
llm_grpo_qwen2_5_math_1_5b_instruct_2n8g_megatrontp1_sglang H100	❌ failed 42 min	❌	❌	Pre-existing on main. Colocated sglang dies at the step-10 async checkpoint on torch.multiprocessing.reductions._rebuild_cuda_tensor_original; main Sept 20/21 fail identically (same step, same attribute) and then sit until the Slurm limit.
llm_grpo_qwen3_30ba3b_4n8g_megatron_qa_nvfp4 H100	❌ failed 39 min	❌	❌	Pre-existing on main. Fake-quant NVFP4 ranks abort at step 1 (Fatal Python error: Aborted, cumem_allocator.cpp on main); main Sept 20 fails identically, Sept 21 bounced at checkout.
llm_grpo_qwen3_5_35ba3b_2n8g_automodel_ep16 H100	❌ failed 27 min	❌	❌	Pre-existing on main. RuntimeError: DeepEP error: timeout (dispatch CPU) at step 1; main Sept 20 and 21 fail identically.
llm_grpo_qwen3_8_27b_2n8g_megatron_tp4pp2cp2 H100	❌ failed 17 min	❌	❌	Pre-existing on main. Dies in transformers at model load; main Sept 20 and 21 fail identically.
llm_ppo_qwen2_5_1_5b_gsm8k_2n8g_megatron_valuetp2sp_dynbatch_noncolocated_async_single_controller H100	❌ failed 40 min	❌	❌	Pre-existing on main. Config validation: SingleController checkpointing with a replay-checkpoint-capable sampler requires checkpointing.save_period; main Sept 20 and 21 fail identically.
llm_prorlv2_qwen2_5_math_1_5b_instruct_1n8g_fsdp2tp1_v2_tq_mooncake H100	❌ failed 185 min	❌	❌	Pre-existing on main. Slurm time limit at step 376/450 (refits healthy); main Sept 20 and 21 hit the same limit.
vlm_vlm_grpo_nemotron_omni_30ba3b_clevr_1n8g_megatron_tp8ep8_v1 H100	❌ failed 36 min	❌	❌	Pre-existing on main. CUDA OOM at step 1; main Sept 20 and 21 fail identically.
vlm_vlm_grpo_nemotron_omni_30ba3b_mmpr_4n8g_megatron_tp8ep16_v1 H100	❌ failed 45 min	❌	❌	Pre-existing on main. AssertionError: defer_fp32_logits must be True if logprob_chunk_size is set; main Sept 20 and 21 fail identically.
vlm_vlm_grpo_qwen3_5_35ba3b_geo3k_2n8g_automodel_ep16 H100	❌ failed 43 min	❌	❌	Pre-existing on main. RuntimeError: DeepEP error: timeout (dispatch CPU) at step 1; main Sept 21 (Sept 20 bounced at checkout) fail identically.
vlm_vlm_grpo_qwen3_5_35ba3b_geo3k_2n8g_automodel_ep16_tq_simple H100	❌ failed 22 min	❌	❌	Pre-existing on main. RuntimeError: DeepEP error: timeout (dispatch CPU) at step 1; main Sept 21 (Sept 20 bounced at checkout) fail identically.

PR passes, main fails (2)

• H100 (1): llm_dapo_gemma4_e2b_it_1n8g_fsdp2_automodel
• GB200 (1): llm_sft_gpt_oss_20b_1n4g_fsdp4ep4_automodel

Pass on both PR and main: 139 (H100 108, GB200 31); the list is in a comment below to keep this description under GitHub's size limit.

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[yfw]

/ok to test 037c678

[yfw]

/ok to test 6ec3dd6

[yfw]

/ok to test fc89153

[yfw]

/ok to test 000cd24

[yfw]

/ok to test d104085

[yfw]

/ok to test 8e97d91

[yfw]

/ok to test 0cc6fab

[yfw]

/ok to test 51d7ee4

[yfw]

/ok to test 5200efa

[yfw]

/ok to test be440de

[copy-pr-bot]

/ok to test be440de

@yfw, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

[yfw]

/ok to test be440de

[yfw]

/ok to test 8d1f98a

[yfw]

/ok to test af7d9e0

[yfw]

Recipes that pass on both this PR (final lock be440de: H100 lane 69151691, GB200 lane 69183689) and on main's Sept 20 / 21 nightlies (139):

H100 (108)

• llm_dapo_gemma4_12b_it_2n8g_fsdp2_automodel
• llm_distillation_qwen3_1_7b_1n8g_megatron_qa_nvfp4
• llm_distillation_qwen3_32b_to_1_7b_base_1n8g_fsdp2tp1_v1
• llm_distillation_qwen3_32b_to_1_7b_base_1n8g_megatron_tp2pp2cp2_pack
• llm_dpo_llama3_1_8b_instruct_4n8g_megatrontp2pp2_quick
• llm_dpo_llama3_2_1b_instruct_1n8g_fsdp2tp1_v2
• llm_gdpo_qwen2_5_1_5b_1n8g_gym_multireward
• llm_grpo_cispo_mm1_async_lag1_highoffpolicy_qwen3_30ba3b_3n8g_megatron_cispo
• llm_grpo_deepscaler_1_5b_16K
• llm_grpo_deepscaler_1_5b_24K
• llm_grpo_deepscaler_1_5b_8K
• llm_grpo_deepscaler_1_5b_8K_tq_simple
• llm_grpo_gemma3_1b_it_1n8g_fsdp2tp1
• llm_grpo_gspo_deepscaler_1_5b_8K
• llm_grpo_gspo_deepscaler_1_5b_8K_tq_simple
• llm_grpo_llama3_1_8b_instruct_1n8g_megatron_fp8_rollouts_v3
• llm_grpo_llama3_1_8b_instruct_1n8g_megatron_fp8_rollouts_v3_tq_simple
• llm_grpo_llama3_1_8b_instruct_2n8g_async_1off_single_controller_streaming2
• llm_grpo_llama3_1_8b_instruct_2n8g_fsdp2tp1_noncolocated
• llm_grpo_llama3_1_8b_instruct_2n8g_fsdp2tp1_noncolocated_tq_simple
• llm_grpo_llama3_1_8b_instruct_2n8g_megatron_fp8_e2e
• llm_grpo_llama3_1_8b_instruct_2n8g_megatron_fp8_e2e_tq_mooncake
• llm_grpo_llama3_2_1b_instruct_1n8g_fsdp2tp1_v3
• llm_grpo_llama3_2_1b_instruct_1n8g_fsdp2tp1_v3_tq_simple
• llm_grpo_llama3_2_1b_instruct_1n8g_fsdp2tp2_temp0_8_topp0_9_topk50
• llm_grpo_llama3_2_1b_instruct_1n8g_megatron
• llm_grpo_llama3_2_1b_instruct_1n8g_megatron_generation
• llm_grpo_llama3_2_1b_instruct_1n8g_megatron_generation_tq_mooncake
• llm_grpo_llama3_2_1b_instruct_1n8g_megatron_temp0_8_topp0_9_topk50
• llm_grpo_llama3_2_1b_instruct_1n8g_megatron_tq_simple
• llm_grpo_math_qwen3_30ba3b_megatron_tp4_32k
• llm_grpo_moonlight_16b_automodel_1n8g_ep8
• llm_grpo_nano_v2_12b_1n8g_megatron
• llm_grpo_nano_v2_12b_2n8g_fsdp2tp1
• llm_grpo_nanov3_30BA3B_1n8g_fsdp2_tq_mooncake_v2
• llm_grpo_nanov3_30BA3B_1n8g_fsdp2_v2
• llm_grpo_nanov3_30BA3B_2n8g_fsdp2_lora
• llm_grpo_nanov3_30BA3B_2n8g_megatron_generation_noncolocated_async_gym
• llm_grpo_nanov3_30BA3B_2n8g_megatron_lora
• llm_grpo_nanov3_30BA3B_2n8g_megatron_pack_cp
• llm_grpo_nanov3_30BA3B_2n8g_megatron_pack_cp_tq_simple
• llm_grpo_nemotron3_super_120BA12B_16n8g_automodel_ep8_v2
• llm_grpo_nemotron3_super_120BA12B_16n8g_megatron
• llm_grpo_qwen2_5_1_5B_4n8g_megatron_yarn_256k
• llm_grpo_qwen2_5_32b_32n8g_fsdp2tp8_actckpt_v3
• llm_grpo_qwen2_5_math_1_5b_instruct_1n8g_fsdp2tp1_v3
• llm_grpo_qwen2_5_math_1_5b_instruct_1n8g_fsdp2tp1_v3_tq_simple
• llm_grpo_qwen2_5_math_1_5b_instruct_1n8g_megatron_generation_noncolocated_single_controller_async_lag4
• llm_grpo_qwen2_5_math_1_5b_instruct_1n8g_megatron_generation_single_controller_async_lag4
• llm_grpo_qwen2_5_math_1_5b_instruct_1n8g_megatron_single_controller_sync
• llm_grpo_qwen2_5_math_1_5b_instruct_1n8g_megatron_single_controller_sync_tq_mooncake
• llm_grpo_qwen2_5_math_1_5b_instruct_2n8g_fsdp2tp1_sglang
• llm_grpo_qwen2_5_math_1_5b_instruct_2n8g_megatrontp1_sglang_noncolocated
• llm_grpo_qwen3_1_7b_1n8g_megatron_eagle3
• llm_grpo_qwen3_1_7b_1n8g_megatron_eagle3_tq_mooncake
• llm_grpo_qwen3_1_7b_1n8g_megatron_super_swe1
• llm_grpo_qwen3_1_7b_2n8g_megatron_super_rlhf
• llm_grpo_qwen3_1_7b_6n8g_megatron_super_rlvr
• llm_grpo_qwen3_30ba3b_10n8g_megatron_cp2_r3_async
• llm_grpo_qwen3_30ba3b_10n8g_megatron_cp2_r3_async_single_controller
• llm_grpo_qwen3_30ba3b_2n8g_megatron_fused_linear_logprobs
• llm_grpo_qwen3_30ba3b_4n8g_megatron_zmq_deltaweight_noncolocated
• llm_grpo_qwen3_30ba3b_8n8g_megatron_cp2_r3
• llm_grpo_qwen3_30ba3b_8n8g_megatron_cp2_r3_tq_simple
• llm_grpo_qwen3_30ba3b_thinking_swe1_16n8g_megatron_cp2_r3_async_gym
• llm_grpo_qwen3_5_35ba3b_2n8g_megatron_ep16tp2_fp8
• llm_grpo_qwen3_5_35ba3b_2n8g_megatron_ep16tp2cp2
• llm_grpo_qwen3_5_9b_1n8g_megatron
• llm_grpo_qwen3_5_9b_1n8g_megatron_fp8
• llm_grpo_qwen3_8B_base_1n8g_fsdp2_lora
• llm_grpo_qwen3_8B_base_1n8g_fsdp2_lora_tq_mooncake
• llm_grpo_qwen3_8b_base_1n8g_fp8_kvcache_megatron
• llm_grpo_qwen3_8b_base_1n8g_megatron_lora
• llm_grpo_qwen3_8b_base_dapo_2n8g_long_megatron_qa_nvfp4_w4a16
• llm_mopd_qwen3_1_7b_3n8g_megatron_pack_single_controller
• llm_mopd_qwen3_1_7b_3n8g_megatron_pack_single_controller_fullvocab
• llm_ppo_qwen2_5_1_5b_gsm8k_1n8g_automodel_noncolocated
• llm_ppo_qwen2_5_1_5b_gsm8k_1n8g_automodel_noncolocated_async
• llm_ppo_qwen2_5_1_5b_gsm8k_1n8g_automodel_valuetp2sp
• llm_ppo_qwen2_5_1_5b_gsm8k_1n8g_megatron_valuetp2sp_dynbatch
• llm_ppo_qwen2_5_1_5b_gsm8k_1n8g_megatron_valuetp2sp_pp2cp2_pack
• llm_ppo_qwen2_5_1_5b_gsm8k_2n8g_megatron_valuetp2sp_dynbatch_noncolocated
• llm_ppo_qwen2_5_1_5b_gsm8k_2n8g_megatron_valuetp2sp_dynbatch_noncolocated_async
• llm_prorlv2_qwen2_5_math_1_5b_instruct_1n8g_fsdp2tp1_v2
• llm_sft_llama3_1_8b_1n8g_fsdp2tp1_lora
• llm_sft_llama3_1_8b_1n8g_fsdp2tp2
• llm_sft_llama3_1_8b_1n8g_fsdp2tp4_dynamicbatch
• llm_sft_llama3_1_8b_1n8g_megatron
• llm_sft_llama3_1_8b_1n8g_megatron_lora
• llm_sft_llama3_1_8b_1n8g_megatron_seqpack
• llm_sft_llama3_2_1b_1n8g_fsdp2tp1_v3
• llm_sft_llama3_2_1b_2n8g_hsdp
• llm_sft_nanov3_30BA3B_2n8g_fsdp2
• llm_sft_nanov3_30BA3B_2n8g_fsdp2_lora
• llm_sft_qwen2_5_32b_4n8g_fsdp2tp8sp_actckpt_v3
• llm_sft_qwen2_5_math7b_1n8g_megatron_fused_linear_logprobs
• llm_sft_qwen2_5_math7b_2n8g_megatron
• vlm_vlm_grpo_gemma4_e4b_geo3k_1n8g_automodel
• vlm_vlm_grpo_nemotron_omni_30ba3b_clevr_1n8g_automodel_ep8_v2
• vlm_vlm_grpo_nemotron_omni_30ba3b_clevr_2n8g_megatron_tp8ep8_v1
• vlm_vlm_grpo_nemotron_omni_30ba3b_clevr_2n8g_megatron_tp8ep8_v1_tq_mooncake
• vlm_vlm_grpo_nemotron_omni_30ba3b_mmpr_4n8g_automodel_ep8_v1
• vlm_vlm_grpo_qwen2_5_omni_3b_avqa_1n8g_megatron_v1
• vlm_vlm_grpo_qwen2_5_omni_7b_audiomcq_1n8g_megatron_v1
• vlm_vlm_grpo_qwen2_5_omni_7b_intent_1n8g_megatron_v1
• vlm_vlm_grpo_qwen2_5_vl_3b_instruct_clevr_1n8g_dtensor2tp1_v1
• vlm_vlm_grpo_qwen2_5_vl_3b_instruct_clevr_1n8g_megatrontp2_v1
• vlm_vlm_grpo_qwen3_omni_30ba3b_audiomcq_4n8g_megatron_v1

GB200 (31)

• llm_distillation_nano3_30ba3b_4n4g_megatron_qa_nvfp4_modelopt_spec
• llm_distillation_qwen3_32b_to_1_7b_base_1n4g_fsdp2tp1_v1
• llm_distillation_qwen3_32b_to_1_7b_base_1n4g_megatron_tp1pp2cp2_pack
• llm_dpo_llama3_1_8b_instruct_4n4g_megatrontp1pp2_quick
• llm_dpo_llama3_2_1b_instruct_1n4g_fsdp2tp1_v2
• llm_grpo_deepscaler_1_5b_1n4g_8K
• llm_grpo_gemma3_1b_it_1n4g_fsdp2tp1
• llm_grpo_llama3_1_8b_instruct_2n4g_async_1off_single_controller_streaming2
• llm_grpo_llama3_1_8b_instruct_2n4g_fsdp2tp1_noncolocated
• llm_grpo_llama3_2_1b_instruct_1n4g_megatron
• llm_grpo_llama3_2_1b_instruct_1n4g_megatron_generation
• llm_grpo_nano_v2_12b_1n4g_megatron
• llm_grpo_nano_v2_12b_2n4g_fsdp2tp1
• llm_grpo_nanov3_30BA3B_4n4g_megatron_generation_noncolocated_async_gym
• llm_grpo_nanov3_30BA3B_4n4g_megatron_generation_noncolocated_mxfp8_rollouts_packed_refit
• llm_grpo_qwen2_5_0_5b_1n4g_megatron_trtllm_noncolocated_async
• llm_grpo_qwen2_5_7b_instruct_4n4g_fsdp2tp2_v3
• llm_grpo_qwen2_5_math_1_5b_instruct_1n4g_fsdp2tp1_v3
• llm_grpo_qwen2_5_math_1_5b_instruct_1n4g_megatron_generation_noncolocated_single_controller_async_lag4
• llm_grpo_qwen2_5_math_1_5b_instruct_1n4g_megatron_generation_single_controller_async_lag4
• llm_grpo_qwen2_5_math_1_5b_instruct_2n4g_megatrontp1_sglang_noncolocated
• llm_grpo_qwen3_1_7b_1n4g_megatron_eagle3
• llm_grpo_qwen3_5_35ba3b_6n4g_async_1off_bf16_trtllm
• llm_sft_llama3_2_1b_1n4g_fsdp2tp1_v3
• llm_sft_nanov3_30BA3B_2n4g_fsdp2
• llm_sft_nanov3_30BA3B_2n4g_fsdp2_lora
• llm_sft_qwen2_5_math7b_2n4g_megatron
• vlm_vlm_grpo_nemotron_omni_30ba3b_clevr_8n4g_megatron_generation_v1
• vlm_vlm_grpo_nemotron_omni_30ba3b_clevr_8n4g_megatron_single_controller_async_v1
• vlm_vlm_grpo_qwen2_5_vl_3b_instruct_clevr_1n4g_dtensor2tp1_v1
• vlm_vlm_grpo_qwen2_5_vl_3b_instruct_clevr_1n4g_megatrontp1_v1

[yfw]

Team review: PR #4080, vLLM 0.26 → 0.29 and torch 2.11 → 2.13

Reviewed at head af7d9e0 by 8 agents (RL codebase, vLLM upstream, torch/packaging, bug finder, design, tests, description/threads, devil's advocate). No correctness bug was found in the code. Inline: one CI-coverage gap that should be fixed here (the three new DCP_USE_PREFIX_STORE tests never run), a handful of low-severity test/doc hardening items with committable blocks, and a description refresh.

Verified against the pinned sources (vLLM v0.29.0 98dff2a8, torch v2.13.0 cf30153c / v2.11.0 70d99e99): all ten source-patch anchors occur exactly once in 0.29 and every patch was executed against a scratch copy of its target file; the TCPStore patch takes the "upstream fixed" branch; all five injected worker extensions are collision-free against Worker/WorkerBase; 239 from vllm... import targets across the repo resolve; check_admission, make_fp8_moe_kernel, make_nvfp4_moe_kernel, _get_tied_embedding_params, SKIP_TENSORS/SKIP_LOAD_TENSORS, the KV-scale mirror and the DeepSeek V4 hook shadow (every exit path) match 0.29 line by line; the tied-alias failure and its fix were reproduced on CPU with real vLLM classes; the amax routing was probed with vLLM's real expert-mapping strings; the CUDA-IPC 'c' payload is byte-identical between the two torch tags and the 2.11 consumer checks version <= own; DCP_USE_PREFIX_STORE and the MASTER_ADDR/MASTER_PORT assert hold in torch 2.13; the lock is below main only for numpy, llguidance and the flash-attn local label (100 packages above, 399 equal).

Local runs (this host has no usable CUDA driver, so GPU-gated tests are unverified). --vllm-only over tests/unit/models/generation in a venv built from this lock's vllm extra: 415 passed, 7 skipped, 8 environment-only failures; every test this PR adds passed. No-vLLM pass (the L0 shards' mode): generation 506 passed, utils 486, data_plane 497. pre-commit run --all-files: clean. 55 mutants were run against the PR's new tests; every pinned fix went red except the vacuities called out inline.

Upstream root causes worth filing (the workarounds are correct, keep them):

• vLLM: DeepseekV4ForCausalLM.load_weights ends with self.process_weights_after_loading() on every call (model.py#L1813-L1816) although the loader already runs the model-level hook (model_loader/utils.py#L145-L146); per-call invocation breaks partial/streamed loads (this PR's Cannot copy out of meta tensor).
• vLLM: AutoWeightsLoader._check_skipped_aliases requires a tied alias and its canonical to arrive in the same load_weights call (utils.py#L442-L470), which no batched weight-update client can guarantee.

FYI, no action. all_gather_single does not exist on torch 2.11, which the sglang/trtllm venvs (also installing nemo_rl) now pin; its only caller BatchedDataDict.all_gather has no production call site, and the rename does remove a per-call FutureWarning on 2.13. check_admission is a no-op today (queue limits default None). The ambiguous-shard detection in quantization/utils.py is generic but only DeepSeek V4 resolves it; four other 0.29 models share the shape and keep the pre-existing bf16-refit fallback (tracking issue at most).

Optional test hardening (verified candidates, not staged): an engine_client member sweep for _AsyncLLMHTTPClient; a WeightsMapper._map_name pin; an IPC tuple-position pin against torch's rebuild_cuda_tensor signature; the mapper hand-off at the tied-alias call sites; RPC caller/callee parity for synchronize_sparse_refit_device; the <= boundary in normalize_cuda_ipc_handle; || true on the nvidia-smi probe in test_decode_vs_prefill.sh; the remaining three injected worker extensions in test_vllm_worker_extension_names.py. Available on request.

Generated by Claude Code

[yfw]

/ok to test e222247

[yfw]

/ok to test ab6b3f2