Skip to content

Add Q2_K reordered MMVQ and ESIMD kernels - #27509

Open
malsbat wants to merge 12 commits into
ggml-org:masterfrom
aicss-genai:esimd-q2k
Open

Add Q2_K reordered MMVQ and ESIMD kernels#27509
malsbat wants to merge 12 commits into
ggml-org:masterfrom
aicss-genai:esimd-q2k

Conversation

@malsbat

@malsbat malsbat commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

This description is taken from the original PR, #26336. The only difference is that this fixes the build error that surfaced post-merge.

Overview

Continuing the work in #25942 to add the Q2_K kernel. In order to support the reordered ESIMD it is necessary to support reordered MMVQ as the MMVQ path is enabled in multi-token calls.

Measured tg improvement of 1.54x to 1.90x (depending on hardware) using Qwen35-27B-Q2_K_M.

Additional information

Note: all measurements were captured with #25741 applied locally as the parent PR does not include the newer upstream fix. This only affects pp numbers which are not the important part of this PR.

PVC performance

Baseline (unreordered MMVQ):

export GGML_SYCL_ENABLE_OPT=0
export GGML_SYCL_PRIORITIZE_DMMV=0
export GGML_SYCL_ENABLE_ESIMD=0
model size params backend ngl sm test t/s
qwen35 27B Q2_K - Medium 11.21 GiB 27.32 B SYCL 100 none pp512 886.33 ± 4.67
qwen35 27B Q2_K - Medium 11.21 GiB 27.32 B SYCL 100 none tg128 11.48 ± 0.00

Reordered MMVQ

export GGML_SYCL_ENABLE_OPT=1
export GGML_SYCL_PRIORITIZE_DMMV=0
export GGML_SYCL_ENABLE_ESIMD=0
model size params backend ngl sm test t/s
qwen35 27B Q2_K - Medium 11.21 GiB 27.32 B SYCL 100 none pp512 883.92 ± 3.91
qwen35 27B Q2_K - Medium 11.21 GiB 27.32 B SYCL 100 none tg128 14.06 ± 0.00

DMMV

export GGML_SYCL_ENABLE_OPT=1
export GGML_SYCL_PRIORITIZE_DMMV=1
export GGML_SYCL_ENABLE_ESIMD=0
model size params backend ngl sm test t/s
qwen35 27B Q2_K - Medium 11.21 GiB 27.32 B SYCL 100 none pp512 888.42 ± 3.82
qwen35 27B Q2_K - Medium 11.21 GiB 27.32 B SYCL 100 none tg128 14.09 ± 0.00

ESIMD

export GGML_SYCL_ENABLE_OPT=1
export GGML_SYCL_PRIORITIZE_DMMV=0
export GGML_SYCL_ENABLE_ESIMD=1
model size params backend ngl sm test t/s
qwen35 27B Q2_K - Medium 11.21 GiB 27.32 B SYCL 100 none pp512 888.90 ± 1.99
qwen35 27B Q2_K - Medium 11.21 GiB 27.32 B SYCL 100 none tg128 17.63 ± 0.13

Perplexity tests across wikitext, hellaswag, and winogrande showed no significant changes between unreordered MMVQ, reordered MMVQ, and ESIMD.

B70 for same 4 scenarios

model size params backend ngl test t/s
qwen35 27B Q2_K - Medium 11.21 GiB 27.32 B SYCL 100 pp512 668.22 ± 1.60
qwen35 27B Q2_K - Medium 11.21 GiB 27.32 B SYCL 100 tg128 10.74 ± 0.03
model size params backend ngl test t/s
qwen35 27B Q2_K - Medium 11.21 GiB 27.32 B SYCL 100 pp512 678.17 ± 1.58
qwen35 27B Q2_K - Medium 11.21 GiB 27.32 B SYCL 100 tg128 15.19 ± 0.07
model size params backend ngl test t/s
qwen35 27B Q2_K - Medium 11.21 GiB 27.32 B SYCL 100 pp512 674.22 ± 1.05
qwen35 27B Q2_K - Medium 11.21 GiB 27.32 B SYCL 100 tg128 17.70 ± 0.09
model size params backend ngl test t/s
qwen35 27B Q2_K - Medium 11.21 GiB 27.32 B SYCL 100 pp512 672.23 ± 1.92
qwen35 27B Q2_K - Medium 11.21 GiB 27.32 B SYCL 100 tg128 20.43 ± 0.13

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES, used Claude heavily to author new kernels based on existing patterns.

malsbat added 12 commits July 20, 2026 21:43
Configure cmake build with -DGGML_SYCL_ESIMD=ON to enable.

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
While not an issue in the current version, this will become an
issue when additional QK ESIMD kernels are added (such as Q2_K).

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
@malsbat
malsbat requested a review from a team as a code owner August 21, 2026 16:43
@github-actions github-actions Bot added ggml changes relating to the ggml tensor library for machine learning SYCL https://en.wikipedia.org/wiki/SYCL - GPU programming language labels Aug 21, 2026
@arthw

arthw commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

@malsbat
The PR #27490 fix the building error and restore the code of #26336 (comment).

After it's merged, could you rebase this PR?

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ggml changes relating to the ggml tensor library for machine learning SYCL https://en.wikipedia.org/wiki/SYCL - GPU programming language

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants