Skip to content

fix(rms_norm): support optional weight - #916

Draft
voltjia wants to merge 1 commit into
masterfrom
fix/rms-norm-optional-weight
Draft

fix(rms_norm): support optional weight#916
voltjia wants to merge 1 commit into
masterfrom
fix/rms-norm-optional-weight

Conversation

@voltjia

@voltjia voltjia commented Aug 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Allow rms_norm to receive an optional weight across the public C++, Python, and legacy C wrappers.
  • Implement weight=None in CPU, shared CUDA, and Cambricon native slot 0 by skipping the affine multiplication.
  • Keep capability differences explicit: NineToothed slot 9 and linked slot 16 reject weight=None before dereferencing it.
  • Preserve the legacy C raw-pointer ABI and translate null weight pointers to an empty optional; add overload-local generator regressions.

Motivation

vLLM v0.26.0 exposes _C::rms_norm(Tensor! result, Tensor input, Tensor? weight, float epsilon) -> (). InfiniOps previously required a concrete weight tensor, preventing normalization without the affine multiply.

Upstream:

Type of Change

  • feat
  • fix
  • perf
  • refactor
  • test
  • docs
  • build / ci
  • chore
  • Breaking change

Platforms Affected

  • CPU (WITH_CPU)
  • NVIDIA (WITH_NVIDIA)
  • Iluvatar (WITH_ILUVATAR)
  • MetaX (WITH_METAX)
  • Cambricon (WITH_CAMBRICON)
  • Moore (WITH_MOORE)
  • Ascend (WITH_ASCEND)
  • PyTorch C++ bindings (WITH_TORCH)
  • Build system / CMake / CI
  • Python bindings / user-facing API

Smoke Test Result

Exact commit: e92fd4df33b857a610aab2cf693688929cd24c27
A100-SXM4-80GB, SM80, CUDA 13.1, PyTorch 2.10.0a0
Clean Release/-O3/-DNDEBUG CPU+NVIDIA build: passed
CI-style smoke (--devices cpu nvidia -n 8): 119 passed, 26 skipped

Test Results on Supported Platforms

Platform Affected Build / Smoke Result Full Result / Notes
CPU Yes A100 validation container passed rms_norm: CPU cases passed; two stream-only cases skipped as expected
NVIDIA Yes Manual clean Release validation passed; GitHub jobs running rms_norm: CPU+NVIDIA 80 passed, 2 skipped; CUDA weighted/None and non-default-stream cases passed
Iluvatar Yes Primary and shadow CI passed Shared CUDA slot 0
MetaX Yes Primary and shadow CI passed Native slot 0 supports None; linked slot 16 rejects it
Cambricon Yes Primary and shadow CI passed Native slot 0 supports None; linked slot 16 rejects it
Moore Yes Primary and shadow CI passed Native slot 0 supports None; linked slot 16 rejects it
Ascend No Primary and shadow CI passed No public Operator<RmsNorm> implementation
Focused validation
tests/test_generate_wrappers.py: 27 passed
tests/test_rms_norm.py --devices cpu nvidia: 80 passed, 2 skipped
Ruff 0.15.22 format/check: passed
Clang Format 21.1.8: passed
git diff/show --check: passed

The two focused skips are the CPU variants of the accelerator-stream tests.

Fresh codegen preserved the legacy C raw descriptor/data pointer signatures. Both descriptor and call paths map weight == nullptr to an empty optional before any dereference.

Benchmark / Performance Impact

N/A. The weighted path keeps the existing kernels; the unweighted path avoids the weight read and multiply.

Notes for Reviewers

  • Tests remain in tests/test_rms_norm.py; no per-provider test module was added.
  • Default dispatch uses native slot 0, so ordinary calls support weight=None.
  • Explicit linked/NineToothed selections reject unsupported None without allocating a unit tensor or falling back.
  • Legacy C signatures remain raw pointer-compatible; null now maps to an empty optional without dereference.
  • The C++ virtual signature changes, so backend implementations must be rebuilt together.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant