From 943555f24b348a9aaa9ccdd4227a5894d4317f8f Mon Sep 17 00:00:00 2001 From: Trawinski Date: Wed, 2 Sep 2026 01:13:14 +0200 Subject: [PATCH 01/10] mtp demo --- .../speculative_decoding/README.md | 265 +++++++++--------- 1 file changed, 126 insertions(+), 139 deletions(-) diff --git a/demos/continuous_batching/speculative_decoding/README.md b/demos/continuous_batching/speculative_decoding/README.md index 92810bf5fc..b58b78b3ab 100644 --- a/demos/continuous_batching/speculative_decoding/README.md +++ b/demos/continuous_batching/speculative_decoding/README.md @@ -6,17 +6,110 @@ OpenVINO GenAI implements three drafting strategies, all exposed through the sam | Strategy | How it drafts | Best for | Extra model required | |---|---|---|---| +| **MTP** | Built-in multi-token prediction head | Models with bundled MTP heads (e.g. Qwen3.8-27B) | No — head bundled with the main model | | **Fast Draft** | Small off-the-shelf LLM | General-purpose; any target/draft pair | Yes — smaller LLM sharing target's tokenizer | | **EAGLE3** | Draft head conditioned on target's hidden states | Highest acceptance rate; code and reasoning; supports tree drafting | Yes — EAGLE3 head trained on the target family | -| **MTP** | Built-in multi-token prediction head | Models with bundled MTP heads (e.g. Qwen3.8-27B) | No — head bundled with the main model | All three strategies share the same server API — only the generation parameters differ. ## Prerequisites -**Model preparation**: Python 3.9 or higher with pip and HuggingFace account +**Model preparation**: Python 3.9 or higher with pip and a Hugging Face account + +**Model Server deployment**: Docker Engine or the OVMS binary package installed according to the [bare-metal deployment guide](../../../docs/deploying_server_baremetal.md) + +# MTP (Multi-Token Prediction) + +MTP replaces the separate draft model with a lightweight prediction head bundled inside the main model weights — no additional download is needed. The head is auto-detected by OVMS when `openvino_mtp_model.xml` is present in the draft model directory. Because it shares the main model's weights, the draft cost is minimal and acceptance rates are high for the same model family. + +## Model considerations + +For this demo we use [OpenVINO/Qwen3.8-27B-int4-ov](https://huggingface.co/OpenVINO/Qwen3.8-27B-int4-ov), which has a bundled MTP head and is exported in INT4 precision. + +> **Note:** This model requires OVMS 2026.4 or weekly pre-release build. See the model card for compatibility details. Prefix caching is not currently supported in MTP mode. + +## Server Deployment + +:::{dropdown} **Deploying with Docker** +```bash +export GPU_ARGS=$(if ls /dev/dri/render* >/dev/null 2>&1; then echo "--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)"; fi) +docker run -d --rm ${GPU_ARGS} -p 8000:8000 -v ${HOME}/models:/models:rw openvino/model_server:weekly \ + --rest_port 8000 \ + --model_repository_path /models \ + --source_model OpenVINO/Qwen3.8-27B-int4-ov \ + --draft_model_path . \ + --enable_prefix_caching false +``` +::: + +:::{dropdown} **Deploying on Bare Metal** +```bat +ovms --rest_port 8000 --model_repository_path c:\models --source_model OpenVINO/Qwen3.8-27B-int4-ov --draft_model_path . --enable_prefix_caching false +``` +::: + +## Request Generation + +The API is identical to other speculative decoding strategies: + +```python +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused") + +response = client.chat.completions.create( + model="OpenVINO/Qwen3.8-27B-int4-ov", + messages=[{"role": "user", "content": "Explain the transformer attention mechanism."}], + temperature=0, + extra_body={"num_assistant_tokens": 5}, +) +print(response.choices[0].message) +``` + +`num_assistant_tokens` controls how many MTP candidates are proposed per target step. The default is `5` if not specified. + +## Check performance + +Check the deployed model's performance by using the vLLM benchmark script and the Sonnet dataset. + +Install vLLM and download the Sonnet dataset: +```bash +pip install vllm --index-url https://wheels.vllm.ai/nightly/cpu --extra-index-url https://pypi.org/simple +curl https://raw.githubusercontent.com/vllm-project/vllm/refs/heads/main/benchmarks/sonnet.txt -o sonnet.txt +``` + +Run benchmark with 100 requests sent sequentially: +```bash +vllm bench serve --dataset-name sonnet --dataset-path sonnet.txt --backend openai-chat --host localhost --port 8000 --endpoint /v3/chat/completions --max-concurrency 1 --model OpenVINO/Qwen3.8-27B-int4-ov --num-prompts 10 +``` +``` +============ Serving Benchmark Result ============ +Successful requests: 10 +Failed requests: 0 +Maximum request concurrency: 1 +Benchmark duration (s): 27.85 +Total input tokens: 5405 +Total generated tokens: 1500 +Request throughput (req/s): 0.36 +Output token throughput (tok/s): 53.86 +Peak output token throughput (tok/s): 72.00 +Peak concurrent requests: 2.00 +Total token throughput (tok/s): 247.95 +---------------Time to First Token---------------- +Mean TTFT (ms): 496.63 +Median TTFT (ms): 433.09 +P99 TTFT (ms): 897.54 +-----Time per Output Token (excl. 1st token)------ +Mean TPOT (ms): 15.35 +Median TPOT (ms): 15.15 +P99 TPOT (ms): 17.88 +---------------Inter-token Latency---------------- +Mean ITL (ms): 16.02 +Median ITL (ms): 0.02 +P99 ITL (ms): 54.90 +================================================== +``` -**Model Server deployment**: Installed Docker Engine or OVMS binary package according to the [baremetal deployment guide](../../../docs/deploying_server_baremetal.md) # EAGLE3 @@ -48,8 +141,12 @@ mkdir models Run `export_model.py` script to download and quantize the model: -```console -python export_model.py text_generation --source_model Qwen/Qwen3-8B --draft_source_model AngelSlim/Qwen3-8B_eagle3 --draft_eagle3_mode --weight-format int4 --config_file_path models/config.json --model_repository_path models +```bat +python export_model.py text_generation --source_model Qwen/Qwen3-8B --draft_source_model AngelSlim/Qwen3-8B_eagle3 --draft_eagle3_mode --weight-format int4 --model_repository_path c:\models +``` +or +```bash +python export_model.py text_generation --source_model Qwen/Qwen3-8B --draft_source_model AngelSlim/Qwen3-8B_eagle3 --draft_eagle3_mode --weight-format int4 --model_repository_path ${HOME}/models ``` Draft model inherits all scheduler properties from the main model. @@ -57,7 +154,6 @@ Draft model inherits all scheduler properties from the main model. You should have a model folder like below: ``` models -├── config.json └── Qwen └── Qwen3-8B ├── added_tokens.json @@ -90,78 +186,24 @@ models :::{dropdown} **Deploying with Docker** ```bash -docker run -d --rm $(test -d /dev/dri && echo "--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)") \ - -p 8000:8000 -v $(pwd)/models:/workspace:ro openvino/model_server:weekly \ - --rest_port 8000 --config_path /workspace/config.json +export GPU_ARGS=$(if ls /dev/dri/render* >/dev/null 2>&1; then echo "--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)"; fi) +docker run ${GPU_ARGS} -d --rm -p 8000:8000 -v ${HOME}/models:/models:ro openvino/model_server:weekly \ + --model_path /models/Qwen/Qwen3-8B \ + --model_name Qwen/Qwen3-8B \ + --rest_port 8000 ``` -OVMS auto-detects the best available device at startup. To target a specific device explicitly, pass `--target_device GPU` (or `NPU`, `HETERO:GPU,CPU`, etc.) to `export_model.py` and follow the [AI accelerators guide](../../../docs/accelerators.md) for additionally required docker parameters. ::: :::{dropdown} **Deploying on Bare Metal** -Assuming you have unpacked model server package, make sure to: - -- **On Windows**: run `setupvars` script -- **On Linux**: set `LD_LIBRARY_PATH` and `PATH` environment variables - -as mentioned in [deployment guide](../../../docs/deploying_server_baremetal.md), in every new shell that will start OpenVINO Model Server. - -Depending on how you prepared models in the first step of this demo, they are deployed to either CPU or GPU (it's defined in `config.json`). If you run on GPU make sure to have appropriate drivers installed, so the device is accessible for the model server. +Install OVMS as described in the [deployment guide](../../../docs/deploying_server_baremetal.md). ```bat -ovms --rest_port 8000 --config_path ./models/config.json +ovms --rest_port 8000 --model_path c:\models\Qwen\Qwen3-8B --model_name Qwen/Qwen3-8B ``` ::: -## Check performance - -Let's check how the deployed model is doing by running performance test. For that purpose we can use vLLM benchmark script and sonnet dataset. - -Install vLLM and download sonnet dataset: -```bash -pip install vllm --index-url https://wheels.vllm.ai/nightly/cpu --extra-index-url https://pypi.org/simple -curl https://raw.githubusercontent.com/vllm-project/vllm/refs/heads/main/benchmarks/sonnet.txt -o sonnet.txt -``` - -Run benchmark with 100 requests sent sequentially: -```bash -vllm bench serve --dataset-name sonnet --dataset-path sonnet.txt --backend openai-chat --host localhost --port 8000 --endpoint /v3/chat/completions --max-concurrency 1 --tokenizer Qwen/Qwen3-8B --model Qwen/Qwen3-8B --num_prompts 100 - -Starting initial single prompt test run... -Skipping endpoint ready check. -Starting main benchmark run... -Traffic request rate: inf -Burstiness factor: 1.0 (Poisson process) -Maximum request concurrency: 1 -100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [06:59<00:00, 4.19s/it] -tip: install termplotlib and gnuplot to plot the metrics -============ Serving Benchmark Result ============ -Successful requests: 100 -Failed requests: 0 -Maximum request concurrency: 1 -Benchmark duration (s): 419.00 -Total input tokens: 54256 -Total generated tokens: 15000 -Request throughput (req/s): 0.24 -Output token throughput (tok/s): 35.80 -Peak output token throughput (tok/s): 16.00 -Peak concurrent requests: 2.00 -Total token throughput (tok/s): 165.29 ----------------Time to First Token---------------- -Mean TTFT (ms): 426.71 -Median TTFT (ms): 424.97 -P99 TTFT (ms): 635.37 ------Time per Output Token (excl. 1st token)------ -Mean TPOT (ms): 25.25 -Median TPOT (ms): 25.09 -P99 TPOT (ms): 29.22 ----------------Inter-token Latency---------------- -Mean ITL (ms): 66.29 -Median ITL (ms): 66.75 -P99 ITL (ms): 72.11 -================================================== -``` ## Chain drafting @@ -184,7 +226,7 @@ print(response.choices[0].message.content) Increase `num_assistant_tokens` until the tokens-per-step figure plateaus, then back off — past the plateau, rejected draft tokens are pure overhead. -Setting `num_assistant_tokens: 0` disables drafting for that request; only the target model runs. +`num_assistant_tokens` must be greater than `0`; OVMS rejects a value of `0` for EAGLE3 decoding. ## Tree drafting @@ -222,10 +264,10 @@ In this demo: - [meta-llama/CodeLlama-7b-hf](https://huggingface.co/meta-llama/CodeLlama-7b-hf) as a main model - [AMD-Llama-135m](https://huggingface.co/amd/AMD-Llama-135m) as a draft model -both in FP16 precision. +both in INT8 precision. ## Model preparation -Here, the original Pytorch LLM models and the tokenizers will be converted to IR format and optionally quantized. +Here, the original PyTorch LLM models and tokenizers are converted to IR format and quantized. That ensures faster initialization time, better performance and lower memory consumption. LLM engine parameters will be defined inside the `graph.pbtxt` file. @@ -238,10 +280,14 @@ mkdir models Run `export_model.py` script to download and quantize the model: -> **Note:** Before downloading the CodeLlama model, access must be requested. Follow the instructions on the [meta-llama/CodeLlama-7b-hf](https://huggingface.co/meta-llama/CodeLlama-7b-hf) to request access. When access is granted, create an authentication token in the HuggingFace account -> Settings -> Access Tokens page. Issue the following command and enter the authentication token. Authenticate via `huggingface-cli login`. +> **Note:** Before downloading the CodeLlama model, request access by following the instructions on the [meta-llama/CodeLlama-7b-hf](https://huggingface.co/meta-llama/CodeLlama-7b-hf) model page. After access is granted, create an authentication token under Hugging Face **Settings > Access Tokens**, run `huggingface-cli login`, and enter the token when prompted. -```console -python export_model.py text_generation --source_model meta-llama/CodeLlama-7b-hf --draft_source_model amd/AMD-Llama-135m --weight-format fp16 --kv_cache_precision u8 --config_file_path models/config.json --model_repository_path models +```bat +python export_model.py text_generation --source_model meta-llama/CodeLlama-7b-hf --draft_source_model amd/AMD-Llama-135m --weight-format int8 --model_repository_path c:\models +``` +or +```bash +python export_model.py text_generation --source_model meta-llama/CodeLlama-7b-hf --draft_source_model amd/AMD-Llama-135m --weight-format int8 --model_repository_path ${HOME}/models ``` Draft model inherits all scheduler properties from the main model. @@ -249,7 +295,6 @@ Draft model inherits all scheduler properties from the main model. You should have a model folder like below: ``` models -├── config.json └── meta-llama └── CodeLlama-7b-hf ├── amd-AMD-Llama-135m @@ -285,25 +330,21 @@ models :::{dropdown} **Deploying with Docker** ```bash -docker run -d --rm -p 8000:8000 -v $(pwd)/models:/workspace:ro openvino/model_server:latest --rest_port 8000 --config_path /workspace/config.json +export GPU_ARGS=$(if ls /dev/dri/render* >/dev/null 2>&1; then echo "--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)"; fi) +docker run -d ${GPU_ARGS} --rm -p 8000:8000 -v ${HOME}/models:/models:ro openvino/model_server:weekly \ + --rest_port 8000 \ + --model_path /models/meta-llama/CodeLlama-7b-hf \ + --model_name meta-llama/CodeLlama-7b-hf ``` -OVMS auto-detects the best available device at startup. To target a specific device explicitly, pass `--target_device GPU` (or `NPU`, `HETERO:GPU,CPU`, etc.) to `export_model.py` and follow the [AI accelerators guide](../../../docs/accelerators.md) for additionally required docker parameters. ::: :::{dropdown} **Deploying on Bare Metal** -Assuming you have unpacked model server package, make sure to: - -- **On Windows**: run `setupvars` script -- **On Linux**: set `LD_LIBRARY_PATH` and `PATH` environment variables - -as mentioned in [deployment guide](../../../docs/deploying_server_baremetal.md), in every new shell that will start OpenVINO Model Server. - -Depending on how you prepared models in the first step of this demo, they are deployed to either CPU or GPU (it's defined in `config.json`). If you run on GPU make sure to have appropriate drivers installed, so the device is accessible for the model server. +Install OVMS as described in the [deployment guide](../../../docs/deploying_server_baremetal.md). ```bat -ovms --rest_port 8000 --config_path ./models/config.json +ovms --rest_port 8000 --model_path c:\models\meta-llama\CodeLlama-7b-hf --model_name meta-llama/CodeLlama-7b-hf ``` ::: @@ -364,61 +405,7 @@ for chunk in stream: `num_assistant_tokens` does not have to be sent on every request — see [Setting default generation parameters](#setting-default-generation-parameters) to configure a deployment-level default. -# MTP (Multi-Token Prediction) - -MTP replaces the separate draft model with a lightweight prediction head bundled inside the main model weights — no additional download is needed. The head is auto-detected by OVMS when `openvino_mtp_model.xml` is present in the draft model directory. Because it shares the main model's weights, the draft cost is minimal and acceptance rates are high for the same model family. - -## Model considerations - -For this demo we use [OpenVINO/Qwen3.8-27B-int8-ov](https://huggingface.co/OpenVINO/Qwen3.8-27B-int8-ov) — a vision-language model with a bundled MTP head exported in INT8 precision. - -> **Note:** This model requires OpenVINO nightly builds and is marked experimental. See the model card for compatibility details. -> **Note:** Prefix caching is not yet supported in this mode. - -## Server Deployment - -:::{dropdown} **Deploying with Docker** -```bash -docker run -d --rm $(test -d /dev/dri && echo "--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)") \ - -p 8000:8000 -v ${HOME}/models:/models:rw openvino/model_server:weekly \ - --rest_port 8000 \ - --model_repository_path /models \ - --source_model OpenVINO/Qwen3.8-27B-int4-ov \ - --draft_model_path . \ - --enable_prefix_caching false -``` -::: - -:::{dropdown} **Deploying on Bare Metal** -```console -ovms --rest_port 8000 \ - --model_repository_path c:\models \ - --source_model OpenVINO/Qwen3.8-27B-int4-ov \ - --draft_model_path . \ - --enable_prefix_caching false -``` -::: - -## Request Generation -The API is identical to other speculative decoding strategies: - -```python -from openai import OpenAI - -client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused") - -response = client.chat.completions.create( - model="OpenVINO/Qwen3.8-27B-int4-ov", - messages=[{"role": "user", "content": "Explain the transformer attention mechanism."}], - temperature=0, - max_tokens=200, - extra_body={"num_assistant_tokens": 5}, -) -print(response.choices[0].message.content) -``` - -`num_assistant_tokens` controls how many MTP candidates are proposed per target step. The default is `5` if not specified. # Setting Default Generation Parameters From 42d1f4d474a2dec4c3207faf4a6290cb6806a3e5 Mon Sep 17 00:00:00 2001 From: Trawinski Date: Wed, 2 Sep 2026 01:34:33 +0200 Subject: [PATCH 02/10] fix --- demos/continuous_batching/speculative_decoding/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/demos/continuous_batching/speculative_decoding/README.md b/demos/continuous_batching/speculative_decoding/README.md index b58b78b3ab..fa42c43b74 100644 --- a/demos/continuous_batching/speculative_decoding/README.md +++ b/demos/continuous_batching/speculative_decoding/README.md @@ -33,7 +33,7 @@ For this demo we use [OpenVINO/Qwen3.8-27B-int4-ov](https://huggingface.co/OpenV :::{dropdown} **Deploying with Docker** ```bash export GPU_ARGS=$(if ls /dev/dri/render* >/dev/null 2>&1; then echo "--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)"; fi) -docker run -d --rm ${GPU_ARGS} -p 8000:8000 -v ${HOME}/models:/models:rw openvino/model_server:weekly \ +docker run -d --rm ${GPU_ARGS} --user $(id -u):$(id -g) -p 8000:8000 -v ${HOME}/models:/models:rw openvino/model_server:weekly \ --rest_port 8000 \ --model_repository_path /models \ --source_model OpenVINO/Qwen3.8-27B-int4-ov \ @@ -187,7 +187,7 @@ models :::{dropdown} **Deploying with Docker** ```bash export GPU_ARGS=$(if ls /dev/dri/render* >/dev/null 2>&1; then echo "--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)"; fi) -docker run ${GPU_ARGS} -d --rm -p 8000:8000 -v ${HOME}/models:/models:ro openvino/model_server:weekly \ +docker run -d ${GPU_ARGS} --user $(id -u):$(id -g) --rm -p 8000:8000 -v ${HOME}/models:/models:ro openvino/model_server:weekly \ --model_path /models/Qwen/Qwen3-8B \ --model_name Qwen/Qwen3-8B \ --rest_port 8000 @@ -331,7 +331,7 @@ models :::{dropdown} **Deploying with Docker** ```bash export GPU_ARGS=$(if ls /dev/dri/render* >/dev/null 2>&1; then echo "--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)"; fi) -docker run -d ${GPU_ARGS} --rm -p 8000:8000 -v ${HOME}/models:/models:ro openvino/model_server:weekly \ +docker run -d ${GPU_ARGS} --user $(id -u):$(id -g) --rm -p 8000:8000 -v ${HOME}/models:/models:ro openvino/model_server:weekly \ --rest_port 8000 \ --model_path /models/meta-llama/CodeLlama-7b-hf \ --model_name meta-llama/CodeLlama-7b-hf From e0774a7e3c574952e0a41d9b23e5a20a333c0db7 Mon Sep 17 00:00:00 2001 From: Trawinski Date: Wed, 2 Sep 2026 01:41:26 +0200 Subject: [PATCH 03/10] dummy change --- src/BUILD | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/BUILD b/src/BUILD index 1e2368327b..97d0d5e00b 100644 --- a/src/BUILD +++ b/src/BUILD @@ -2939,4 +2939,4 @@ ovms_cc_library( "//src:libovms_ov_utils",], visibility = ["//visibility:public"], alwayslink = 1, -) +) \ No newline at end of file From c0210212cd9d9153d013b9fa09a670d1c1cf2d5a Mon Sep 17 00:00:00 2001 From: Trawinski Date: Wed, 2 Sep 2026 09:03:02 +0200 Subject: [PATCH 04/10] set max_tokens --- demos/continuous_batching/speculative_decoding/README.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/demos/continuous_batching/speculative_decoding/README.md b/demos/continuous_batching/speculative_decoding/README.md index fa42c43b74..18777f6f84 100644 --- a/demos/continuous_batching/speculative_decoding/README.md +++ b/demos/continuous_batching/speculative_decoding/README.md @@ -26,7 +26,7 @@ MTP replaces the separate draft model with a lightweight prediction head bundled For this demo we use [OpenVINO/Qwen3.8-27B-int4-ov](https://huggingface.co/OpenVINO/Qwen3.8-27B-int4-ov), which has a bundled MTP head and is exported in INT4 precision. -> **Note:** This model requires OVMS 2026.4 or weekly pre-release build. See the model card for compatibility details. Prefix caching is not currently supported in MTP mode. +> **Note:** This model requires OVMS 2026.4 or weekly pre-release build. See the model card for compatibility details. Prefix caching is not currently supported in MTP mode. It is also required to set max_tokens parameter ## Server Deployment @@ -61,6 +61,7 @@ response = client.chat.completions.create( model="OpenVINO/Qwen3.8-27B-int4-ov", messages=[{"role": "user", "content": "Explain the transformer attention mechanism."}], temperature=0, + max_tokens=10000, extra_body={"num_assistant_tokens": 5}, ) print(response.choices[0].message) @@ -218,7 +219,7 @@ response = client.chat.completions.create( model="Qwen/Qwen3-8B", messages=[{"role": "user", "content": "What is OpenVINO?"}], temperature=0, - max_tokens=200, + max_tokens=2000, extra_body={"num_assistant_tokens": 5}, ) print(response.choices[0].message.content) @@ -390,7 +391,7 @@ stream = client.completions.create( model="meta-llama/CodeLlama-7b-hf", prompt="def quicksort(numbers):", temperature=0, - max_tokens=100, + max_tokens=2000, extra_body={"num_assistant_tokens": 5}, stream=True, ) From a08392bc4a19730678dadcde0391102018e83490 Mon Sep 17 00:00:00 2001 From: "Trawinski, Dariusz" Date: Wed, 2 Sep 2026 11:57:06 +0200 Subject: [PATCH 05/10] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- demos/continuous_batching/speculative_decoding/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/demos/continuous_batching/speculative_decoding/README.md b/demos/continuous_batching/speculative_decoding/README.md index 18777f6f84..59e770c5a4 100644 --- a/demos/continuous_batching/speculative_decoding/README.md +++ b/demos/continuous_batching/speculative_decoding/README.md @@ -227,7 +227,7 @@ print(response.choices[0].message.content) Increase `num_assistant_tokens` until the tokens-per-step figure plateaus, then back off — past the plateau, rejected draft tokens are pure overhead. -`num_assistant_tokens` must be greater than `0`; OVMS rejects a value of `0` for EAGLE3 decoding. +Setting `num_assistant_tokens: 0` disables drafting for that request; only the target model runs. ## Tree drafting From 99bf4365bc569b3396e1efa7a38f5e7c05cb36e3 Mon Sep 17 00:00:00 2001 From: Trawinski Date: Wed, 2 Sep 2026 23:26:15 +0200 Subject: [PATCH 06/10] instal opoenai --- demos/continuous_batching/speculative_decoding/README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/demos/continuous_batching/speculative_decoding/README.md b/demos/continuous_batching/speculative_decoding/README.md index 18777f6f84..1dcdb7192e 100644 --- a/demos/continuous_batching/speculative_decoding/README.md +++ b/demos/continuous_batching/speculative_decoding/README.md @@ -51,7 +51,9 @@ ovms --rest_port 8000 --model_repository_path c:\models --source_model OpenVINO/ ## Request Generation The API is identical to other speculative decoding strategies: - +```console +pip install openai +``` ```python from openai import OpenAI From 6a3b1e130dfe0b6867238709b9e51832e5778c1b Mon Sep 17 00:00:00 2001 From: Trawinski Date: Thu, 3 Sep 2026 11:41:06 +0200 Subject: [PATCH 07/10] test --- demos/common/export_models/requirements.txt | 2 +- demos/continuous_batching/speculative_decoding/README.md | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/demos/common/export_models/requirements.txt b/demos/common/export_models/requirements.txt index 54c76788cf..f567b2137d 100644 --- a/demos/common/export_models/requirements.txt +++ b/demos/common/export_models/requirements.txt @@ -1,7 +1,7 @@ --extra-index-url "https://download.pytorch.org/whl/cpu" --extra-index-url "https://storage.openvinotoolkit.org/simple/wheels/nightly" --extra-index-url "https://storage.openvinotoolkit.org/simple/wheels/pre-release" -optimum-intel@git+https://github.com/huggingface/optimum-intel.git@fd9499006ef4c3f35d5c6647851b2e9fa8711912 +optimum-intel accelerate datasets diffusers # for image generation diff --git a/demos/continuous_batching/speculative_decoding/README.md b/demos/continuous_batching/speculative_decoding/README.md index 7a7d505fe3..29b5cee3b8 100644 --- a/demos/continuous_batching/speculative_decoding/README.md +++ b/demos/continuous_batching/speculative_decoding/README.md @@ -63,7 +63,6 @@ response = client.chat.completions.create( model="OpenVINO/Qwen3.8-27B-int4-ov", messages=[{"role": "user", "content": "Explain the transformer attention mechanism."}], temperature=0, - max_tokens=10000, extra_body={"num_assistant_tokens": 5}, ) print(response.choices[0].message) @@ -236,6 +235,9 @@ Setting `num_assistant_tokens: 0` disables drafting for that request; only the t Tree drafting adds two `GenerationConfig` fields. Setting `tree_depth > 0` switches from chain to tree mode: ```python +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:8000/v3", api_key="unused") response = client.chat.completions.create( model="Qwen/Qwen3-8B", messages=[{"role": "user", "content": "What is OpenVINO?"}], From 8045ba01733a22247982706803127ac77c279a61 Mon Sep 17 00:00:00 2001 From: Trawinski Date: Thu, 3 Sep 2026 11:49:57 +0200 Subject: [PATCH 08/10] WA for github --- ci/build_test_OnCommit.groovy | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/ci/build_test_OnCommit.groovy b/ci/build_test_OnCommit.groovy index 5573d55ce6..526535c770 100644 --- a/ci/build_test_OnCommit.groovy +++ b/ci/build_test_OnCommit.groovy @@ -38,6 +38,13 @@ pipeline { agent { label 'ovmsbuilder' } + environment { + // Force HTTP/1.1: some proxies mangle git's HTTP/2 smart-protocol responses, + // causing "fatal: expected flush after ref listing" on fetch. + GIT_CONFIG_COUNT = "1" + GIT_CONFIG_KEY_0 = "http.version" + GIT_CONFIG_VALUE_0 = "HTTP/1.1" + } options { timeout(time: 4, unit: 'HOURS') } From 3b629156b4baf158f17e6295e73c1e25909b210c Mon Sep 17 00:00:00 2001 From: Trawinski Date: Thu, 3 Sep 2026 14:27:44 +0200 Subject: [PATCH 09/10] tune parameters --- .../speculative_decoding/README.md | 170 +----------------- 1 file changed, 8 insertions(+), 162 deletions(-) diff --git a/demos/continuous_batching/speculative_decoding/README.md b/demos/continuous_batching/speculative_decoding/README.md index 29b5cee3b8..b02cd93bba 100644 --- a/demos/continuous_batching/speculative_decoding/README.md +++ b/demos/continuous_batching/speculative_decoding/README.md @@ -82,7 +82,7 @@ curl https://raw.githubusercontent.com/vllm-project/vllm/refs/heads/main/benchma Run benchmark with 100 requests sent sequentially: ```bash -vllm bench serve --dataset-name sonnet --dataset-path sonnet.txt --backend openai-chat --host localhost --port 8000 --endpoint /v3/chat/completions --max-concurrency 1 --model OpenVINO/Qwen3.8-27B-int4-ov --num-prompts 10 +vllm bench serve --dataset-name sonnet --dataset-path sonnet.txt --backend openai-chat --host localhost --port 8000 --endpoint /v1/chat/completions --max-concurrency 1 --model OpenVINO/Qwen3.8-27B-int4-ov --num-prompts 10 ``` ``` ============ Serving Benchmark Result ============ @@ -214,7 +214,7 @@ Send `num_assistant_tokens` to control how many candidates the draft head propos ```python from openai import OpenAI -client = OpenAI(base_url="http://localhost:8000/v3", api_key="unused") +client = OpenAI(base_url="http://localhost:8000/1", api_key="unused") response = client.chat.completions.create( model="Qwen/Qwen3-8B", @@ -237,179 +237,25 @@ Tree drafting adds two `GenerationConfig` fields. Setting `tree_depth > 0` switc ```python from openai import OpenAI -client = OpenAI(base_url="http://localhost:8000/v3", api_key="unused") +client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused") response = client.chat.completions.create( model="Qwen/Qwen3-8B", messages=[{"role": "user", "content": "What is OpenVINO?"}], temperature=0, - max_tokens=200, + max_tokens=2000, extra_body={ - "num_assistant_tokens": 15, # candidates verified per step - "branching_factor": 8, # top-k expansions per tree layer - "tree_depth": 4, # draft head iterations + "num_assistant_tokens": 5, # candidates verified per step + "branching_factor": 4, # top-k expansions per tree layer + "tree_depth": 2, # draft head iterations }, ) +print(response.choices[0].message.content) ``` `total_draft_tokens = branching_factor² × (tree_depth − 1) + branching_factor` must be ≥ `num_assistant_tokens`. A reasonable starting point is `branching_factor=4..8`, `tree_depth=3..4`. Tree drafting is EAGLE3-only; it cannot be combined with beam search or multinomial sampling. -# Fast Draft - -Fast Draft is the classic two-model setup: a smaller off-the-shelf LLM that shares the target's tokenizer proposes tokens autoregressively, and the target model verifies them. It works with any target/draft pair without retraining. The speedup depends on how often the small model's distribution agrees with the large one. - -## Model considerations - -Both models must share the same tokenizer so draft token IDs map correctly to target token IDs. - -Performance gain depends heavily on the model pair and workload — the optimal combination should be found empirically. Model sizes and precisions both factor in. - -In this demo: - - [meta-llama/CodeLlama-7b-hf](https://huggingface.co/meta-llama/CodeLlama-7b-hf) as a main model - - [AMD-Llama-135m](https://huggingface.co/amd/AMD-Llama-135m) as a draft model - -both in INT8 precision. - -## Model preparation -Here, the original PyTorch LLM models and tokenizers are converted to IR format and quantized. -That ensures faster initialization time, better performance and lower memory consumption. -LLM engine parameters will be defined inside the `graph.pbtxt` file. - -Download export script, install its dependencies and create directory for the models: -```console -curl https://raw.githubusercontent.com/openvinotoolkit/model_server/refs/heads/main/demos/common/export_models/export_model.py -o export_model.py -pip3 install -r https://raw.githubusercontent.com/openvinotoolkit/model_server/refs/heads/main/demos/common/export_models/requirements.txt -mkdir models -``` - -Run `export_model.py` script to download and quantize the model: - -> **Note:** Before downloading the CodeLlama model, request access by following the instructions on the [meta-llama/CodeLlama-7b-hf](https://huggingface.co/meta-llama/CodeLlama-7b-hf) model page. After access is granted, create an authentication token under Hugging Face **Settings > Access Tokens**, run `huggingface-cli login`, and enter the token when prompted. - -```bat -python export_model.py text_generation --source_model meta-llama/CodeLlama-7b-hf --draft_source_model amd/AMD-Llama-135m --weight-format int8 --model_repository_path c:\models -``` -or -```bash -python export_model.py text_generation --source_model meta-llama/CodeLlama-7b-hf --draft_source_model amd/AMD-Llama-135m --weight-format int8 --model_repository_path ${HOME}/models -``` - -Draft model inherits all scheduler properties from the main model. - -You should have a model folder like below: -``` -models -└── meta-llama - └── CodeLlama-7b-hf - ├── amd-AMD-Llama-135m - │   ├── config.json - │   ├── generation_config.json - │   ├── openvino_detokenizer.bin - │   ├── openvino_detokenizer.xml - │   ├── openvino_model.bin - │   ├── openvino_model.xml - │   ├── openvino_tokenizer.bin - │   ├── openvino_tokenizer.xml - │   ├── special_tokens_map.json - │   ├── tokenizer_config.json - │   ├── tokenizer.json - │   └── tokenizer.model - ├── config.json - ├── generation_config.json - ├── graph.pbtxt - ├── openvino_detokenizer.bin - ├── openvino_detokenizer.xml - ├── openvino_model.bin - ├── openvino_model.xml - ├── openvino_tokenizer.bin - ├── openvino_tokenizer.xml - ├── special_tokens_map.json - ├── tokenizer_config.json - ├── tokenizer.json - └── tokenizer.model - -``` - -## Server Deployment - -:::{dropdown} **Deploying with Docker** -```bash -export GPU_ARGS=$(if ls /dev/dri/render* >/dev/null 2>&1; then echo "--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)"; fi) -docker run -d ${GPU_ARGS} --user $(id -u):$(id -g) --rm -p 8000:8000 -v ${HOME}/models:/models:ro openvino/model_server:weekly \ - --rest_port 8000 \ - --model_path /models/meta-llama/CodeLlama-7b-hf \ - --model_name meta-llama/CodeLlama-7b-hf -``` - -::: - -:::{dropdown} **Deploying on Bare Metal** - -Install OVMS as described in the [deployment guide](../../../docs/deploying_server_baremetal.md). - -```bat -ovms --rest_port 8000 --model_path c:\models\meta-llama\CodeLlama-7b-hf --model_name meta-llama/CodeLlama-7b-hf -``` -::: - -## Readiness Check - -Wait for the model to load. You can check the status with a simple command: -```console -curl http://localhost:8000/v1/config -``` -```json -{ - "meta-llama/CodeLlama-7b-hf": { - "model_version_status": [ - { - "version": "1", - "state": "AVAILABLE", - "status": { - "error_code": "OK", - "error_message": "OK" - } - } - ] - } -} -``` - -## Request Generation - -Models used in this demo — `meta-llama/CodeLlama-7b-hf` and `AMD-Llama-135m` — are base (non-chat) models, so we use the `completions` endpoint. - -```console -pip3 install openai -``` -```python -from openai import OpenAI - -client = OpenAI( - base_url="http://localhost:8000/v3", - api_key="unused" -) - -stream = client.completions.create( - model="meta-llama/CodeLlama-7b-hf", - prompt="def quicksort(numbers):", - temperature=0, - max_tokens=2000, - extra_body={"num_assistant_tokens": 5}, - stream=True, -) -for chunk in stream: - if chunk.choices[0].text is not None: - print(chunk.choices[0].text, end="", flush=True) -``` - -**`num_assistant_tokens`** controls how many tokens the draft model proposes before the main model validates them. High values pay off when the draft frequently agrees with the target; low values reduce wasted work when it doesn't. `5` is a good starting point. - -**`assistant_confidence_threshold`** is an alternative stopping criterion: the draft keeps proposing while its token probability exceeds the threshold, then hands off to the target. It is mutually exclusive with `num_assistant_tokens`. Supported on the Continuous Batching backend only (`LM_CB`) — the stateful backend (`pipeline_type: LM`) does not implement dynamic-length drafting. - -`num_assistant_tokens` does not have to be sent on every request — see [Setting default generation parameters](#setting-default-generation-parameters) to configure a deployment-level default. - # Setting Default Generation Parameters From 9854b28a9c6cbab124a7025acf8370ca60654625 Mon Sep 17 00:00:00 2001 From: Trawinski Date: Thu, 3 Sep 2026 14:29:19 +0200 Subject: [PATCH 10/10] tune --- demos/continuous_batching/speculative_decoding/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/demos/continuous_batching/speculative_decoding/README.md b/demos/continuous_batching/speculative_decoding/README.md index b02cd93bba..2cec1e9e11 100644 --- a/demos/continuous_batching/speculative_decoding/README.md +++ b/demos/continuous_batching/speculative_decoding/README.md @@ -61,7 +61,7 @@ client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused") response = client.chat.completions.create( model="OpenVINO/Qwen3.8-27B-int4-ov", - messages=[{"role": "user", "content": "Explain the transformer attention mechanism."}], + messages=[{"role": "user", "content": "Explain briefly the transformer attention mechanism."}], temperature=0, extra_body={"num_assistant_tokens": 5}, )