Add SGLang Kimi-K3 for MI355X - #183
Open
coketaste wants to merge 2 commits into
Open
Conversation
Add pyt_sglang_kimi-k3 and pyt_sglang_kimi-k3_dspark benchmarking moonshotai/Kimi-K3 online serving, mirroring the vLLM K3 entry added in a40063d. Both track the AMD day-0 recipes in sgl-project/sglang#32548; K3 support itself is sgl-project/sglang#32541. Two entries rather than one because the two published recipes differ only in speculative decoding (DSPARK with the RadixArk/Kimi-K3-DSpark draft) and are reported as separate rows in the tracking issue, so both are worth tracking. They share a model repo, which is why the runner gains a --variant filter: the existing --model filter cannot select between them. Add scripts/sglang/run_sglang.py and configs/, a config-driven serving runner structured after scripts/vllm/run_vllm.py and emitting the same perf CSV schema, so multiple_results ingestion is shared. The existing scripts/sglang/run.sh is left untouched; it is bound to the offline getopts path and hard-exits on non-gfx94* GPUs. Add docker/pyt_sglang_kimi_k3 pinned to lmsysorg/sglang-rocm:rocm720-mi35x-k3-20260727, the day-0 image the issue's MI355X tables were measured against, rather than retargeting docker/pyt_sglang, which is still on lmsysorg/sglang:v0.4.5-rocm630 and would move the existing SGLang entry onto a pre-release base. A newer -20260728 tag exists but nothing published ties it to the recipe, so it is deliberately not adopted. Fold back once K3 lands in a versioned ROCm image. TP8 gfx950 only, hence skip_gpu_arch gfx942. Both are left out of any sweep tag so a tag run does not pull the checkpoint and hold 8 GPUs; they are invocable explicitly by name. Three deviations from the recipe are intentional. The runner emits --tp-size where the recipe writes --tp: there is no --tp server argument, tp_size declares the single alias --tensor-parallel-size, and --tp resolves only through argparse prefix matching, which any future --tp* option would silently break. The server-start timeout is raised to 5400s from run_vllm.py's 1800s, which that commit already flagged as tight for a checkpoint this size. There is no gsm8k accuracy stage, for the same reason as the vLLM entry: K3's always-on reasoning is returned inline over /v1/completions and exhausts the hardcoded 2048-token budget. The issue does not state its input and output lengths. They were recovered from its own tables, where (E2EL - TTFT) / TPOT + 1 lands on ~1024 output tokens on every row and concurrency x (inp + out) / E2EL reproduces the reported total throughput only at 8192 input tokens; keeping that shape is what makes these numbers comparable to the issue's. --random-range-ratio is pinned to 1.0 because it defaults to 0.0, which would randomize lengths down to a single token. Verified statically only, no hardware run. Config expansion yields 5 configs per variant and the variant filter separates them; an end-to-end dry run with subprocess stubbed produced the expected server and benchmark argv, both hf download calls, and correctly ignored a stale leading line in the appended JSONL output. All 14 server flags and 11 benchmark flags were validated against #32541's head branch rather than sglang main, which matters: the PR is still open and kimi_k3 is absent from main's reasoning DetectorMap, so the parser names and DSPARK are only valid inside the pinned image. Synthetic results were fed through madengine's handle_multiple_results, and filter_images_by_skip_gpu_arch skips both entries on gfx942 and runs them on gfx950 with no existing entry changed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Blueprints table in the root README is the entry point to each framework's landing page, so the SGLang row's Models column should name K3 now that both entries exist. The landing page itself, benchmark/sglang/README.md, was updated in ede3844. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
coketaste
requested review from
Rohan138,
amathews-amd,
gargrahul and
ppalaniappan-amd
as code owners
July 28, 2026 23:01
Contributor
There was a problem hiding this comment.
Pull request overview
Adds new MAD benchmarks for SGLang online serving of moonshotai/Kimi-K3 on MI355X (gfx950/TP8), including a DSpark speculative-decoding variant, using a dedicated ROCm day-0 Docker image and a new config-driven serving runner.
Changes:
- Introduces
scripts/sglang/run_sglang.py(config-driven SGLang online serving benchmark runner) andscripts/sglang/configs/kimi_k3.yaml(nospec + dspark variants). - Adds
scripts/sglang/run_kimi_k3.shentrypoint and registers two new model entries inmodels.json. - Adds a dedicated Dockerfile pinned to the MI35x K3 day-0 image and updates docs/README to list Kimi-K3 under SGLang.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/sglang/run_sglang.py | New config-driven SGLang online serving runner producing MAD multiple-results CSV output. |
| scripts/sglang/run_kimi_k3.sh | Entry script to execute the Kimi-K3 serving benchmark and move perf CSV to parent dir. |
| scripts/sglang/configs/kimi_k3.yaml | Serving sweep configuration for Kimi-K3 nospec/dspark variants (8192/1024, conc 2–32). |
| models.json | Adds two new SGLang Kimi-K3 benchmark entries (nospec + dspark) gated off gfx942. |
| docker/pyt_sglang_kimi_k3.ubuntu.amd.Dockerfile | Dedicated ROCm day-0 SGLang image pinned for Kimi-K3 support on MI35x. |
| benchmark/sglang/README.md | Documents the special-case Kimi-K3 entries, invocation, and rationale for settings. |
| README.md | Adds Kimi-K3 to the top-level supported SGLang models list. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+256
to
+258
| # only reported under speculative decoding | ||
| if output.get("accept_length"): | ||
| metrics["accept_length"] = str(output["accept_length"]) |
Comment on lines
+274
to
+281
| finally: | ||
| # kill server and children | ||
| parent = psutil.Process(server.pid) | ||
| for child in parent.children(recursive=True): | ||
| child.send_signal(signal.SIGINT) | ||
| server.send_signal(signal.SIGINT) | ||
| _ = server.communicate() | ||
| del server |
Comment on lines
+145
to
+148
| # split config into common args and list args | ||
| common_cfgs = {k: v for k, v in cfg.items() if k not in SUPPORTED_LIST_ARGS} | ||
| list_cfgs = {k: str(v).split(' ') for k, v in cfg.items() if k in SUPPORTED_LIST_ARGS} | ||
| # expand list args into one dict per combination |
Comment on lines
+59
to
+64
| python3 -u run_sglang.py --config $CONFIG --model $MODEL --variant $VARIANT | ||
|
|
||
| # move the output csv to parent directory | ||
| MODEL_NAME=$(basename $MODEL) | ||
| OUTPUT_CSV="perf_${MODEL_NAME}.csv" | ||
| mv $OUTPUT_CSV ../ |
Comment on lines
+49
to
+50
| # record configuration for posterity | ||
| RUN pip3 list |
coketaste
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
pyt_sglang_kimi-k3andpyt_sglang_kimi-k3_dspark, benchmarkingmoonshotai/Kimi-K3 online serving with SGLang on MI355X. Both track the AMD
day-0 recipes in sgl-project/sglang#32548 (day-0 support: sgl-project/sglang#32541).
speculative decoding (DSPARK with the RadixArk/Kimi-K3-DSpark draft) and are
reported as separate rows in the tracking issue. They share a model repo, so
the runner gains a
--variantfilter to select between them.scripts/sglang/run_sglang.py+scripts/sglang/configs/kimi_k3.yaml:a config-driven online serving runner structured after
scripts/vllm/run_vllm.py, emitting the same perf CSV schema somultiple_resultsingestion is shared. The existingscripts/sglang/run.sh(offlinebench_one_batch/bench_offline_throughputpath, gated to gfx94x) is untouched.
docker/pyt_sglang_kimi_k3.ubuntu.amd.Dockerfile, pinned tolmsysorg/sglang-rocm:rocm720-mi35x-k3-20260727, the day-0 image theissue's MI355X tables were measured against — kept separate from
docker/pyt_sglang(still onlmsysorg/sglang:v0.4.5-rocm630) so theexisting SGLang entry doesn't move onto a pre-release base. Fold back once
K3 lands in a versioned ROCm image.
skip_gpu_arch: gfx942). Both entries are left out of anysweep tag so a tag run does not pull the ~1.56 TB checkpoint and hold 8
GPUs; they're invocable explicitly by name.
Deviations from the recipe (intentional)
--tp-size, not the recipe's--tp: there is no--tpserverargument,
tp_sizedeclares the single alias--tensor-parallel-size, and--tponly resolves through argparse prefix matching, which any future--tp*option would silently break.SGLANG_SERVER_START_TIMEOUT) fromrun_vllm.py's 1800s, which is tight for a checkpoint this size./v1/completionsand exhausts the hardcoded 2048-token budget, making thescore meaningless (same reasoning as the sibling vLLM K3 entry).
recovered from its own tables —
(E2EL - TTFT) / TPOT + 1lands on ~1024output tokens on every row, and
concurrency * (inp + out) / E2ELreproduces the reported total throughput only at 8192 input tokens.
--random-range-ratiois pinned to 1.0 to keep that shape (it defaults to0.0, which randomizes lengths down to a single token).
Verification
Verified statically only, no hardware run:
expand_configs) yields 5 configs per variant, and the--variantfilter separatesnospec/dsparkcorrectly.[Kimi] Support kimi-k3 sgl-project/sglang#32541's head branch (not
main—kimi_k3is absentfrom
main's reasoningDetectorMap, so the parser names and DSPARKsupport only exist inside the pinned image).
sglang.benchmark.serving's CLI flags, output JSON keys, and append-mode--output-filebehavior were checked against the upstream source, matchingwhat
run_sglang.pyexpects.multiple_resultsingestion.filter_images_by_skip_gpu_archskips both new entries on gfx942 and runsthem on gfx950, with no existing entry affected.
Test plan
pyt_sglang_kimi-k3on an 8x MI355X node and confirm the serverstarts and
perf_Kimi-K3.csvis produced with the expected metrics.pyt_sglang_kimi-k3_dsparkand confirm theaccept_lengthcolumnis populated.