Conversation
There was a problem hiding this comment.
🟡 Changes recommended
The updated README contains several factual/instructional inconsistencies (e.g., benchmark request counts, EAGLE3 num_assistant_tokens=0 behavior, and inconsistent model directory paths) that should be corrected before merge.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Updates the speculative decoding continuous batching demo documentation to add an MTP walkthrough and refresh deployment/export instructions for speculative decoding strategies.
Changes:
- Adds a dedicated MTP (Multi-Token Prediction) section with deployment, request, and benchmarking examples.
- Refreshes EAGLE3 and Fast Draft sections (export commands, deployment examples, and some explanatory text/output).
- Reorders/clarifies the drafting strategy comparison table and prerequisites.
File summaries
| File | Description |
|---|---|
| demos/continuous_batching/speculative_decoding/README.md | Adds MTP demo content and updates speculative decoding setup/deployment/benchmarking instructions for multiple strategies. |
Review details
Suppressed comments (3)
demos/continuous_batching/speculative_decoding/README.md:190
- This Docker example mounts
${HOME}/models, but the earlier steps in this section create./modelsand the directory tree example is rooted atmodels/. To keep the instructions consistent, mount the localmodelsdirectory instead.
docker run ${GPU_ARGS} -d --rm -p 8000:8000 -v ${HOME}/models:/models:ro openvino/model_server:weekly \
demos/continuous_batching/speculative_decoding/README.md:290
- Earlier in this section the instructions create a local
modelsdirectory and the folder tree example is rooted atmodels/, but this command writes to${HOME}/models. Consider using--model_repository_path modelsto match the preceding steps and the shown directory layout.
python export_model.py text_generation --source_model meta-llama/CodeLlama-7b-hf --draft_source_model amd/AMD-Llama-135m --weight-format int8 --model_repository_path ${HOME}/models
demos/continuous_batching/speculative_decoding/README.md:334
- This Docker example mounts
${HOME}/models, but the earlier steps in this section create./modelsand the directory tree example is rooted atmodels/. To keep the instructions consistent, mount the localmodelsdirectory instead.
docker run -d ${GPU_ARGS} --rm -p 8000:8000 -v ${HOME}/models:/models:ro openvino/model_server:weekly \
- Files reviewed: 1/1 changed files
- Comments generated: 3
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| curl https://raw.githubusercontent.com/vllm-project/vllm/refs/heads/main/benchmarks/sonnet.txt -o sonnet.txt | ||
| ``` | ||
|
|
||
| Run benchmark with 100 requests sent sequentially: |
| ``` | ||
| or | ||
| ```bash | ||
| python export_model.py text_generation --source_model Qwen/Qwen3-8B --draft_source_model AngelSlim/Qwen3-8B_eagle3 --draft_eagle3_mode --weight-format int4 --model_repository_path ${HOME}/models |
| ```bat | ||
| python export_model.py text_generation --source_model Qwen/Qwen3-8B --draft_source_model AngelSlim/Qwen3-8B_eagle3 --draft_eagle3_mode --weight-format int4 --model_repository_path c:\models | ||
| ``` | ||
| or | ||
| ```bash | ||
| python export_model.py text_generation --source_model Qwen/Qwen3-8B --draft_source_model AngelSlim/Qwen3-8B_eagle3 --draft_eagle3_mode --weight-format int4 --model_repository_path ${HOME}/models |
There was a problem hiding this comment.
Essentially the same. Maybe we should note which command is supposed to use when. Like simple headers: Linux, Windows etc.?
| docker run -d ${GPU_ARGS} --user $(id -u):$(id -g) --rm -p 8000:8000 -v ${HOME}/models:/models:ro openvino/model_server:weekly \ | ||
| --model_path /models/Qwen/Qwen3-8B \ | ||
| --model_name Qwen/Qwen3-8B \ | ||
| --rest_port 8000 |
There was a problem hiding this comment.
Missing draft model. This command is not a speculative decoding pipeline deployment.
|
|
||
| ```bat | ||
| ovms --rest_port 8000 --config_path ./models/config.json | ||
| ovms --rest_port 8000 --model_path c:\models\Qwen\Qwen3-8B --model_name Qwen/Qwen3-8B |
There was a problem hiding this comment.
Same as above. This command runs OVMS with regular Qwen3-8B model, not a speculative decoding pipeline.
| Increase `num_assistant_tokens` until the tokens-per-step figure plateaus, then back off — past the plateau, rejected draft tokens are pure overhead. | ||
|
|
||
| Setting `num_assistant_tokens: 0` disables drafting for that request; only the target model runs. | ||
| `num_assistant_tokens` must be greater than `0`; OVMS rejects a value of `0` for EAGLE3 decoding. |
There was a problem hiding this comment.
Please revert. New message is incorrect as pointed by copilot.
| docker run -d ${GPU_ARGS} --user $(id -u):$(id -g) --rm -p 8000:8000 -v ${HOME}/models:/models:ro openvino/model_server:weekly \ | ||
| --rest_port 8000 \ | ||
| --model_path /models/meta-llama/CodeLlama-7b-hf \ | ||
| --model_name meta-llama/CodeLlama-7b-hf |
There was a problem hiding this comment.
Not a speculative decoding deployment. Add draft model parameters.
|
|
||
| ```bat | ||
| ovms --rest_port 8000 --config_path ./models/config.json | ||
| ovms --rest_port 8000 --model_path c:\models\meta-llama\CodeLlama-7b-hf --model_name meta-llama/CodeLlama-7b-hf |
There was a problem hiding this comment.
Not a speculative decoding deployment. Add draft model parameters.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
🛠 Summary
🧪 Checklist
``