[Qwen3.8] Support Qwen3.8 on branch v0.5.15

[apinge]

Motivation

Support Qwen3.8-2.4T-A95B-FP8 on this v0.5.15 branch. This change partially references the implementation in branch qwen3_8_mi308x

Modifications

Change the backbone from Qwen3_5ForCausalLM to Qwen3_5Model to allow the Qwen3.8 text-only checkpoint to be initialized directly as a causal LM.

Accuracy Tests

  export GLOO_SOCKET_IFNAME=<ETH_IFACE>
  export NCCL_SOCKET_IFNAME=<ETH_IFACE>
  export RCCL_SOCKET_IFNAME=<ETH_IFACE>

  export NCCL_NET=IB
  export NCCL_IB_HCA=<IB_HCA_0>,<IB_HCA_1>,...
  export NCCL_IB_GID_INDEX=<GID_INDEX>
  export NCCL_SOCKET_FAMILY=AF_INET
  export RCCL_AINIC_ROCE=1
 
  export SGLANG_USE_AITER=1

  export NCCL_DEBUG=WARN
  export NCCL_MAX_NCHANNELS=16
  export NCCL_MIN_NCHANNELS=8
  export NCCL_BUFFSIZE=1048576

  python3 -m sglang.launch_server \
    --model-path <QWEN3_8_MODEL_PATH> \
    --tp-size 8 \
    --pp-size 2 \
    --nnodes 2 \
    --node-rank <0_OR_1> \
    --dist-init-addr <RANK0_IP>:25000 \
    --dist-timeout 3600 \
    --attention-backend aiter \
    --chunked-prefill-size 8192 \
    --watchdog-timeout 1200 \
    --mem-fraction-static 0.95 \
    --host 0.0.0.0 \
    --port 9001 \
    --disable-radix-cache \
    --disable-custom-all-reduce \
    --page-size 16 \
    --kv-cache-dtype fp8_e4m3 \
    --cuda-graph-max-bs 8 \
    --max-running-requests 16 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder

lm_eval --model local-completions \
    --model_args '{"base_url":"http://localhost:8001/v1/completions","model":"/models/Qwen3.8-2.4T-A95B-FP8","num_concurrent":256,"max_retries":10,"max_gen_toks":2048,"max_length":8192,"tokenized_requests":false,"timeout":3600}' \
    --tasks gsm8k \
    --num_fewshot 5 \
    --trust_remote_code

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.9515	±	0.0059
		strict-match	5	exact_match	↑	0.9507	±	0.0060

Speed Tests and Profiling

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

[Copilot]

Pull request overview

Adds Qwen3.8 text-only support on v0.5.15 through Qwen3.5-compatible model changes and ROCm RMSNorm paths.

Changes:

• Registers text-only Qwen configurations.
• Refactors dense/MoE model support and weight loading.
• Adds ROCm Triton Gemma RMSNorm dispatch.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Findings
python/sglang/srt/utils/hf_transformers/common.py	Critical import failure because text config classes are not re-exported.
python/sglang/srt/models/qwen3_5.py	Critical incompatibility with the MTP integration and MoE configuration handling.
python/sglang/srt/layers/layernorm.py	Critical AITER ABI mismatch; moderate empty-batch and batch-invariant handling issues.

Suppressed comments (7)

python/sglang/srt/models/qwen3_5.py:1741

• Fused checkpoint names retain a suffix such as .weight (and FP8 scale variants), so these checks never match keys like ...experts.gate_up_proj.weight: the string does not end with experts.gate_up_proj. is_fused_expert consequently stays false, causing packed routed-expert tensors to bypass splitting/distribution; the remapped shared slot experts.{num_experts}.gate_up_proj.* is missed as well. Match the base substrings and the remapped shared slot explicitly.

                if name.endswith("experts.gate_up_proj") or name.endswith(
                    "experts.down_proj"
                ):

python/sglang/srt/models/qwen3_5.py:1889

• With pipeline parallelism, make_layers leaves PPMissingLayer placeholders outside this rank's range, and those placeholders have no mlp. When EPLB evaluates this lazy property, the comprehension therefore raises AttributeError instead of returning the local expert weights. Iterate only over start_layer:end_layer and guard the mlp lookup.

        self._routed_experts_weights_of_layer = LazyValue(
            lambda: {
                layer_id: layer.mlp.get_moe_weights()
                for layer_id, layer in enumerate(self.model.layers)
                if isinstance(layer.mlp, Qwen2MoeSparseMoeBlock)

python/sglang/srt/models/qwen3_5.py:1383

• For MoE inputs this coercion discards the original model_type: Qwen3_5MoeTextConfig becomes Qwen3_5TextConfig. The MTP implementation passes the target text config to this class and has explicit MoE expert loading, but the resulting Qwen3_5Model now builds dense MLPs instead of Qwen2MoeSparseMoeBlock, so MoE MTP weights cannot load or execute. Derive moe from the input model type or preserve the subclass.

        config = _coerce_qwen3_5_text_config(config, moe=False)

python/sglang/srt/models/qwen3_5.py:1382

• qwen3_5_mtp.py:99-103 still constructs this class with is_nextn=True, but the new constructor no longer accepts that argument. Every Qwen3.5 MTP model therefore raises TypeError during initialization. Preserve the parameter and forward it to Qwen3_5Model, which already supports it.

    ) -> None:

python/sglang/srt/models/qwen3_5.py:1389

• This wrapper changes the parameter namespace from the old body-only layout to model.*. Existing body-only callers do not account for that extra nesting: MTP rewrites mtp.layers.* to model.layers.* (qwen3_5_mtp.py:287-290), and MiniCPM strips model.language_model. to layers.* (minicpmv.py:1581-1584). Consequently their LLM weights no longer match params_dict and remain at initialization values. Keep the backbone layout compatible or update both callers' name remapping together with this wrapper.

        self.model = Qwen3_5Model(
            config, quant_config, prefix=add_prefix("model", prefix)

python/sglang/srt/models/qwen3_5.py:1447

• This new forward applies the LM head and returns LogitsProcessorOutput, but the class is still used as a body-only model by both qwen3_5_mtp.py:197-208 and MiniCPMV4_6.forward (minicpmv.py:1446-1457), which each apply their own logits processor. Those callers will either process logits twice or fail when treating the result as hidden states. Keep this class body-only for those integrations (and let the outer model own the head), or switch both callers to Qwen3_5Model and update their loading paths.

        return self.logits_processor(
            input_ids, hidden_states, self.lm_head, forward_batch, aux_hidden_states
        )

python/sglang/srt/models/qwen3_5.py:1405

• The new text-only wrapper owns capture_aux_hidden_states, but set_dflash_layers_to_capture was left on Qwen3_5Model and is not delegated here. ModelRunner.init_aux_hidden_state_capture() calls this method on the registered top-level model for DFLASH (model_runner.py:1055-1061), so enabling DFLASH with a text-only Qwen3.5 model now raises the missing-method ValueError. Add a wrapper method that sets the flag and delegates to self.model.

        self.capture_aux_hidden_states = False

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[apinge]

modification in qwen3_5.py

Before: qwen3_5.py only exposed VL entry classes; the class named``Qwen3_5ForCausalLMwas actually used as the language-model backbone inside the VL wrapper. This PR splits the backbone intoQwen3_5Model` / `Qwen3_5MoeModel` and adds real text-only CausalLM wrappers, enabling Qwen3.8 text-only MoE checkpoints.

• Before qwen3.5.py only exposed VL entry class:

     Qwen3_5ForCausalLM
      - Acts as backbone
      - contains: embed_tokens / layers / norm
      - forward returns hidden_states
      - __init__supports is_nextn
    
    Qwen3_5MoeForCausalLM
      - Inherits Qwen3_5ForCausalLM
      - Acts as MoE backbone
      - forward still returns hidden_states
    
    Qwen3_5ForConditionalGeneration
      - Inherits Qwen3VLForConditionalGeneration
      - Qwen3VLForConditionalGeneration contains:
          visual
          model = Qwen3_5ForCausalLM  # Also used as backbone here
          lm_head
          logits_processor
      - outer forward runs visual/model first then computes logits
    
    Qwen3_5MoeForConditionalGeneration
      - Inherits Qwen3VLForConditionalGeneration
      - contains:
          visual
          model = Qwen3_5MoeForCausalLM  # Also used as backbone here
          lm_head
          logits_processor
    
    Qwen3_5ForCausalLMMTP
      - MTP wrapper
      - contains:
          fc
          pre_fc_norm_embedding / pre_fc_norm_hidden
          model = Qwen3_5ForCausalLM(is_nextn=True)  # Also used as backbone here
          lm_head
          logits_processor
      - forward expects self.model(...) to return hidden_states, then computes logits itself
      ```
    
    - After this PR splits the backbone into `Qwen3_5Model` / `Qwen3_5MoeModel` and adds real text-only CausalLM
      wrappers, enabling Qwen3.8 text-only MoE checkpoints.
  

      Qwen3_5Model
        - backbone
        - Inherits: nn.Module
        - contains: embed_tokens / layers / norm
        - forward return hidden_states
        - __init__ support is_nextn
        -  Same role as the old  Qwen3_5ForCausalLM
      
      Qwen3_5MoeModel
        - Inherits Qwen3_5Model
        - moe backbone
        - No extra logic at present; mainly for wrapper selection
      
      Qwen3_5ForCausalLM
        - Newly added text-only dense causal LM wrapper
        - Inherits: nn.Module
        - contains:
            model = Qwen3_5Model
            lm_head
            logits_processor
        - forward return logits processor output
        - no is_nextn
      
      Qwen3_5MoeForCausalLM
        - New/updated text-only MoE causal LM wrapper
        - Inherits: Qwen3_5ForCausalLM
        -  However __init__ does not call Qwen3_5ForCausalLM.__init__，instead it constructs MoE wrapper directly
        - contains:
            model = Qwen3_5MoeModel
            lm_head
            logits_processor
        - forward return logits processor output
        - Also retains the weight loading logic for MoE/shared expert
      
      Qwen3_5ForConditionalGeneration
        - Still VL outer wrapper
        - Inherits: Qwen3VLForConditionalGeneration
        - contains:
            visual
            model = Qwen3_5Model
            lm_head
            logits_processor
        -  The backbone is passed here, not the Qwen3_5ForCausalLM wrapper
      
      Qwen3_5MoeForConditionalGeneration
        - Still VL MoE outer wrapper
        - Inherits: Qwen3VLForConditionalGeneration
        - contains:
            visual
            model = Qwen3_5MoeModel
            lm_head
            logits_processor
        -  Still The backbone is passed here, not the Qwen3_5ForCausalLM wrapper
      
      Qwen3_5ForCausalLMMTP
        - Still MTP outer wrapper
        - Inherits: nn.Module
        - contains:
            fc
            pre_fc_norm_embedding / pre_fc_norm_hidden
            model = Qwen3_5Model(is_nextn=True)
            lm_head
            logits_processor
        - forward still expects self.model(...) return hidden_states

Existing Qwen3.5 dense/moe checkpoints continue to resolve to Qwen3_5ForConditionalGeneration / Qwen3_5MoeForConditionalGeneration, so the existing VL/ConditionalGeneration path is preserved.

Qwen3.8 uses the text-only moe path and therefore resolves to Qwen3_5MoeForCausalLM .

[apinge]

@sammysun0711 About gemma rmsnorm I profiled current version.
In prefill , qwen3.8 uses same kernel as qwen3.5

In decode, because --disable-custom-all-reduce is set, there is no rmsnorm fusion

The profile is 8k in 1conc.