Problem
MTP prompt prefill appears to lose substantial throughput because target-model prefill and MTP-cache population are interleaved after every prompt chunk.
I am testing Qwen3.8-27B-exl3-7.00bpw with native MTP on two GPUs (RTX 4070 Ti SUPER 16 GB + RTX 5060 Ti 16 GB), with a 131072-token Q4 cache.
Stock EXL3 effectively performs:
target chunk 1
MTP prefill chunk 1
target chunk 2
MTP prefill chunk 2
...
At 124,928 input tokens, stock MTP repeatedly gives about 529-530 input tokens/s.
For comparison, the same model without MTP previously reached about 1000 input tokens/s, so the MTP prefill penalty becomes particularly large at long context.
Instrumentation suggests that the MTP layer's actual prefill computation is not itself large enough to explain this. With forced CUDA synchronization on a 16K prompt I measured approximately:
target MTP-aware forward: 17.54 s
MTP draft prefill: 0.68 s
Normal target prefill under the same forced synchronization took about 16.73 s.
This led me to test whether interrupting target prefill after every chunk is itself causing avoidable scheduling/synchronization overhead.
Solution
As a proof of concept, I deferred MTP-cache population for several target-prefill chunks.
Instead of:
target 1 -> MTP 1
target 2 -> MTP 2
target 3 -> MTP 3
I queue the original input IDs, shifted target hidden states and original cache position for several chunks:
target 1
target 2
...
target N
then:
MTP 1
MTP 2
...
MTP N
The queue is then synchronized and cleared, so memory usage remains bounded.
This produced a clear improvement as the defer depth increased.
124,928 input / ~1,024 output:
Stock:
529-530 input t/s
Depth 4:
711 input t/s
Depth 6:
738 input t/s
Depth 8:
758 input t/s
So depth 8 improved long-context prompt throughput by approximately 43% over stock MTP.
Wall time also improved substantially:
Stock: 4:40.9 and 4:46.1
Depth 4: 3:41.2
Depth 6: 3:34.1
Depth 8: 3:25.4
Short-context testing showed the same effect.
16,384 input / ~1,024 output:
Stock MTP:
831 input t/s
31.4 output t/s
53.0 s
Deferred depth 4, run 1:
1077 input t/s
29.8 output t/s
50.3 s
Deferred depth 4, run 2:
1049 input t/s
31.9 output t/s
48.5 s
So short-context prefill improved by roughly 28% as well.
A bounded deferred MTP-prefill queue therefore seems worth considering in the scheduler.
Alternatives
I considered several alternatives while investigating this.
-
Changing the MTP draft cache from Q4 to Q8 made performance substantially worse, so draft-cache quantization does not appear to explain the problem.
-
N-gram speculation preserves fast prompt processing but only improved generation slightly, so it does not provide the same benefit as MTP.
-
Fully deferring all MTP prefill until target prefill has finished also worked in a 16K test, but retaining all hidden states on GPU would require roughly 1.2-1.3 GB for a ~125K prompt. I assume this is faster, but could be an expensive way to gain a few percentage points. This could alternatively be spilled to pinned system RAM, but I have not tested that approach.
The bounded queue seems like the least invasive option tested so far. At a 3072-token chunk size and hidden size 5120, each retained FP16 hidden-state chunk is roughly 30 MiB, so depth 8 needs only around 240 MiB of temporary storage regardless of total context length.
Explanation
The current per-chunk MTP prefill scheduling appears to leave a substantial amount of prompt-processing performance unused on multi-GPU setups.
In my tests, changing only the scheduling of MTP cache population improved long-context prompt throughput from about 530 t/s to 758 t/s, approximately +43%, while keeping speculative acceptance in the same normal range.
The same effect also appears at shorter context: 16K prompt throughput improved from 831 t/s to roughly 1050-1080 t/s.
This matters because MTP is otherwise a strong generation optimisation, but its current prompt-prefill penalty can outweigh much of that benefit on large prompts. A bounded deferred queue recovers a large portion of that lost throughput with relatively modest temporary memory use.
The improvement also scales consistently with queue depth in my tests:
stock: ~530 t/s
depth 4: 711 t/s
depth 6: 738 t/s
depth 8: 758 t/s
That makes this look less like benchmark noise and more like a scheduling opportunity worth investigating.
I am not suggesting that my proof-of-concept implementation should be merged as-is. The main reason for raising this is to show that there appears to be significant performance available by reducing how often target prefill is interrupted by MTP cache population.
Examples
Test model:
Qwen3.8-27B-exl3-7.00bpw
Backend:
TabbyAPI / ExLlamaV3
Target GPUs:
RTX 4070 Ti SUPER 16 GB
RTX 5060 Ti 16 GB
Relevant config:
max_seq_len: 131072
cache_size: 131072
cache_mode: Q4
tensor_parallel: false
gpu_split_auto: false
gpu_split: [14.5, 13.5]
draft_model:
draft_mode: mtp
dynamic_draft: true
draft_cache_mode: Q4
draft_num_tokens: 7
Prompt chunk size:
3072 tokens
124,928 input / ~1,024 output
Stock MTP run 1:
Input: 529 t/s
Output: 26.0 t/s
Wall: 4:40.9
Acceptance: 53.41%
Stock MTP run 2:
Input: 530 t/s
Output: 22.9 t/s
Wall: 4:46.1
Acceptance: 47.63%
Deferred depth 4:
Input: 711 t/s
Output: 25.6 t/s
Wall: 3:41.2
Acceptance: 50.75%
Deferred depth 6:
Input: 738 t/s
Output: 26.1 t/s
Wall: 3:34.1
Acceptance: 52.92%
Deferred depth 8:
Input: 758 t/s
Output: 29.2 t/s
Wall: 3:25.4
16,384 input / ~1,024 output
Stock MTP:
Input: 831 t/s
Output: 31.4 t/s
Wall: 53.0 s
Deferred depth 4, run 1:
Input: 1077 t/s
Output: 29.8 t/s
Wall: 50.3 s
Deferred depth 4, run 2:
Input: 1049 t/s
Output: 31.9 t/s
Wall: 48.5 s
The experimental scheduling change was approximately:
Stock:
target chunk 1
MTP chunk 1
target chunk 2
MTP chunk 2
...
Deferred:
target chunk 1
target chunk 2
...
target chunk N
MTP chunk 1
MTP chunk 2
...
MTP chunk N
repeat
The queue stores:
prefill_ids.clone()
shifted_hidden
prefill_start
and when the queue reaches the selected depth, the queued MTP prefills are run in order using their original cache positions.
Pseudo-code:
queue.append((
prefill_ids.clone(),
shifted_hidden,
prefill_start,
))
if len(queue) >= DEFER_DEPTH:
for ids, hidden, start in queue:
self.generator.draft_model.prefill(
input_ids=ids,
params={
"target_hidden": hidden,
"attn_mode": "flash_attn",
"block_table": seq.block_index_tensor,
"cache": self.generator.draft_cache,
"cache_seqlens": torch.tensor(
[start],
dtype=torch.int32,
),
"indexed_embeddings": self.embeddings,
},
)
for dev in range(torch.cuda.device_count()):
torch.cuda.synchronize(dev)
queue.clear()
Any remaining queued chunks are flushed, with synchronization, when prompt prefill completes.
Approximate temporary hidden-state storage for Qwen3.8-27B at 3072-token chunks:
depth 4: ~120 MiB
depth 6: ~180 MiB
depth 8: ~240 MiB
Additional context
A few notes from the investigation:
-
I initially suspected the MTP draft prefill itself was the main cost, but instrumentation did not support that. With explicit CUDA synchronization on a 16K prompt, the target-model MTP-aware forward took ~17.54 s while the MTP draft prefill itself took only ~0.68 s.
-
Normal target prefill under the same forced synchronization took ~16.73 s. This suggests the large real-world MTP prefill penalty is not simply due to the arithmetic cost of the extra MTP layer.
-
The improvement from deferred batching was monotonic in the long-context test:
stock ~530 t/s
depth 4 711 t/s
depth 6 738 t/s
depth 8 758 t/s
-
Speculative acceptance remained in the same broad range between stock and deferred runs, so I have not seen evidence that deferred cache construction is degrading draft quality.
-
I also tested changing the MTP draft cache from Q4 to Q8. That made both prompt processing and generation slower, so cache quantization did not appear to be the source of the issue.
-
N-gram speculation was also tested as a possible middle ground. It preserved fast prefill but only gave a small generation improvement compared with MTP, so it did not solve the same problem.
-
I have only tested this on my two-GPU setup and with Qwen3.8-27B. I have not tested multi-sequence jobs, multimodal prompts, recurrent-model edge cases, or all prompt-cache reuse scenarios.
-
The proof-of-concept patch is intentionally crude. It duplicates flush logic, uses explicit device-wide CUDA synchronization, and stores deferred hidden states on GPU. I would expect an upstream-quality implementation to structure this differently.
-
A full-prompt defer also worked at 16K, but bounded batching is more practical because it keeps temporary VRAM usage fixed. At depth 8 with 3072-token chunks, the retained FP16 hidden-state storage is only around 240 MiB.
The main reason for opening the issue is not to propose this exact implementation, but to show that there seems to be a reproducible scheduling opportunity in how MTP prompt cache population is interleaved with target prefill.
Acknowledgements
Problem
MTP prompt prefill appears to lose substantial throughput because target-model prefill and MTP-cache population are interleaved after every prompt chunk.
I am testing Qwen3.8-27B-exl3-7.00bpw with native MTP on two GPUs (RTX 4070 Ti SUPER 16 GB + RTX 5060 Ti 16 GB), with a 131072-token Q4 cache.
Stock EXL3 effectively performs:
target chunk 1
MTP prefill chunk 1
target chunk 2
MTP prefill chunk 2
...
At 124,928 input tokens, stock MTP repeatedly gives about 529-530 input tokens/s.
For comparison, the same model without MTP previously reached about 1000 input tokens/s, so the MTP prefill penalty becomes particularly large at long context.
Instrumentation suggests that the MTP layer's actual prefill computation is not itself large enough to explain this. With forced CUDA synchronization on a 16K prompt I measured approximately:
target MTP-aware forward: 17.54 s
MTP draft prefill: 0.68 s
Normal target prefill under the same forced synchronization took about 16.73 s.
This led me to test whether interrupting target prefill after every chunk is itself causing avoidable scheduling/synchronization overhead.
Solution
As a proof of concept, I deferred MTP-cache population for several target-prefill chunks.
Instead of:
target 1 -> MTP 1
target 2 -> MTP 2
target 3 -> MTP 3
I queue the original input IDs, shifted target hidden states and original cache position for several chunks:
target 1
target 2
...
target N
then:
MTP 1
MTP 2
...
MTP N
The queue is then synchronized and cleared, so memory usage remains bounded.
This produced a clear improvement as the defer depth increased.
124,928 input / ~1,024 output:
Stock:
529-530 input t/s
Depth 4:
711 input t/s
Depth 6:
738 input t/s
Depth 8:
758 input t/s
So depth 8 improved long-context prompt throughput by approximately 43% over stock MTP.
Wall time also improved substantially:
Stock: 4:40.9 and 4:46.1
Depth 4: 3:41.2
Depth 6: 3:34.1
Depth 8: 3:25.4
Short-context testing showed the same effect.
16,384 input / ~1,024 output:
Stock MTP:
831 input t/s
31.4 output t/s
53.0 s
Deferred depth 4, run 1:
1077 input t/s
29.8 output t/s
50.3 s
Deferred depth 4, run 2:
1049 input t/s
31.9 output t/s
48.5 s
So short-context prefill improved by roughly 28% as well.
A bounded deferred MTP-prefill queue therefore seems worth considering in the scheduler.
Alternatives
I considered several alternatives while investigating this.
Changing the MTP draft cache from Q4 to Q8 made performance substantially worse, so draft-cache quantization does not appear to explain the problem.
N-gram speculation preserves fast prompt processing but only improved generation slightly, so it does not provide the same benefit as MTP.
Fully deferring all MTP prefill until target prefill has finished also worked in a 16K test, but retaining all hidden states on GPU would require roughly 1.2-1.3 GB for a ~125K prompt. I assume this is faster, but could be an expensive way to gain a few percentage points. This could alternatively be spilled to pinned system RAM, but I have not tested that approach.
The bounded queue seems like the least invasive option tested so far. At a 3072-token chunk size and hidden size 5120, each retained FP16 hidden-state chunk is roughly 30 MiB, so depth 8 needs only around 240 MiB of temporary storage regardless of total context length.
Explanation
The current per-chunk MTP prefill scheduling appears to leave a substantial amount of prompt-processing performance unused on multi-GPU setups.
In my tests, changing only the scheduling of MTP cache population improved long-context prompt throughput from about 530 t/s to 758 t/s, approximately +43%, while keeping speculative acceptance in the same normal range.
The same effect also appears at shorter context: 16K prompt throughput improved from 831 t/s to roughly 1050-1080 t/s.
This matters because MTP is otherwise a strong generation optimisation, but its current prompt-prefill penalty can outweigh much of that benefit on large prompts. A bounded deferred queue recovers a large portion of that lost throughput with relatively modest temporary memory use.
The improvement also scales consistently with queue depth in my tests:
stock: ~530 t/s
depth 4: 711 t/s
depth 6: 738 t/s
depth 8: 758 t/s
That makes this look less like benchmark noise and more like a scheduling opportunity worth investigating.
I am not suggesting that my proof-of-concept implementation should be merged as-is. The main reason for raising this is to show that there appears to be significant performance available by reducing how often target prefill is interrupted by MTP cache population.
Examples
Test model:
Qwen3.8-27B-exl3-7.00bpw
Backend:
TabbyAPI / ExLlamaV3
Target GPUs:
RTX 4070 Ti SUPER 16 GB
RTX 5060 Ti 16 GB
Relevant config:
max_seq_len: 131072
cache_size: 131072
cache_mode: Q4
tensor_parallel: false
gpu_split_auto: false
gpu_split: [14.5, 13.5]
draft_model:
draft_mode: mtp
dynamic_draft: true
draft_cache_mode: Q4
draft_num_tokens: 7
Prompt chunk size:
3072 tokens
124,928 input / ~1,024 output
Stock MTP run 1:
Input: 529 t/s
Output: 26.0 t/s
Wall: 4:40.9
Acceptance: 53.41%
Stock MTP run 2:
Input: 530 t/s
Output: 22.9 t/s
Wall: 4:46.1
Acceptance: 47.63%
Deferred depth 4:
Input: 711 t/s
Output: 25.6 t/s
Wall: 3:41.2
Acceptance: 50.75%
Deferred depth 6:
Input: 738 t/s
Output: 26.1 t/s
Wall: 3:34.1
Acceptance: 52.92%
Deferred depth 8:
Input: 758 t/s
Output: 29.2 t/s
Wall: 3:25.4
16,384 input / ~1,024 output
Stock MTP:
Input: 831 t/s
Output: 31.4 t/s
Wall: 53.0 s
Deferred depth 4, run 1:
Input: 1077 t/s
Output: 29.8 t/s
Wall: 50.3 s
Deferred depth 4, run 2:
Input: 1049 t/s
Output: 31.9 t/s
Wall: 48.5 s
The experimental scheduling change was approximately:
Stock:
target chunk 1
MTP chunk 1
target chunk 2
MTP chunk 2
...
Deferred:
target chunk 1
target chunk 2
...
target chunk N
MTP chunk 1
MTP chunk 2
...
MTP chunk N
repeat
The queue stores:
prefill_ids.clone()
shifted_hidden
prefill_start
and when the queue reaches the selected depth, the queued MTP prefills are run in order using their original cache positions.
Pseudo-code:
queue.append((
prefill_ids.clone(),
shifted_hidden,
prefill_start,
))
if len(queue) >= DEFER_DEPTH:
for ids, hidden, start in queue:
self.generator.draft_model.prefill(
input_ids=ids,
params={
"target_hidden": hidden,
"attn_mode": "flash_attn",
"block_table": seq.block_index_tensor,
"cache": self.generator.draft_cache,
"cache_seqlens": torch.tensor(
[start],
dtype=torch.int32,
),
"indexed_embeddings": self.embeddings,
},
)
Any remaining queued chunks are flushed, with synchronization, when prompt prefill completes.
Approximate temporary hidden-state storage for Qwen3.8-27B at 3072-token chunks:
depth 4: ~120 MiB
depth 6: ~180 MiB
depth 8: ~240 MiB
Additional context
A few notes from the investigation:
I initially suspected the MTP draft prefill itself was the main cost, but instrumentation did not support that. With explicit CUDA synchronization on a 16K prompt, the target-model MTP-aware forward took ~17.54 s while the MTP draft prefill itself took only ~0.68 s.
Normal target prefill under the same forced synchronization took ~16.73 s. This suggests the large real-world MTP prefill penalty is not simply due to the arithmetic cost of the extra MTP layer.
The improvement from deferred batching was monotonic in the long-context test:
stock ~530 t/s
depth 4 711 t/s
depth 6 738 t/s
depth 8 758 t/s
Speculative acceptance remained in the same broad range between stock and deferred runs, so I have not seen evidence that deferred cache construction is degrading draft quality.
I also tested changing the MTP draft cache from Q4 to Q8. That made both prompt processing and generation slower, so cache quantization did not appear to be the source of the issue.
N-gram speculation was also tested as a possible middle ground. It preserved fast prefill but only gave a small generation improvement compared with MTP, so it did not solve the same problem.
I have only tested this on my two-GPU setup and with Qwen3.8-27B. I have not tested multi-sequence jobs, multimodal prompts, recurrent-model edge cases, or all prompt-cache reuse scenarios.
The proof-of-concept patch is intentionally crude. It duplicates flush logic, uses explicit device-wide CUDA synchronization, and stores deferred hidden states on GPU. I would expect an upstream-quality implementation to structure this differently.
A full-prompt defer also worked at 16K, but bounded batching is more practical because it keeps temporary VRAM usage fixed. At depth 8 with 3072-token chunks, the retained FP16 hidden-state storage is only around 240 MiB.
The main reason for opening the issue is not to propose this exact implementation, but to show that there seems to be a reproducible scheduling opportunity in how MTP prompt cache population is interleaved with target prefill.
Acknowledgements