vulkan: gfx1151 K-quant int-mmq large tile 128x256 @ 512 threads (~1.8x dense prefill) - #5
Open
aic0d3r wants to merge 2 commits into
Open
vulkan: gfx1151 K-quant int-mmq large tile 128x256 @ 512 threads (~1.8x dense prefill)#5aic0d3r wants to merge 2 commits into
aic0d3r wants to merge 2 commits into
Conversation
The dense integer-dot mmq path picks its large tile when both matmul dims exceed 64, which is every big GEMM in a dense-model prefill. On gfx1151 (RDNA3 iGPU, e.g. Radeon 8060S) the stock 128x128 tile at 256 threads underfills the 40 CUs at prefill shapes and re-reads the A weights once per 128-token slab; doubling only BN (and the thread count, keeping the per-thread register tiling identical) halves the A re-reads per token. Measured on Qwen3.8-27B UD-Q5_K_XL-v2, bundled RADV, -fa 1 -b/-ub 2048, same binary interleaved A/B with the tile forced back via GGML_VK_MMQ_INT_K: pp4096 527.4 +-3.1 t/s vs 289.0 +-0.8 (1.82x) pp512 588.7 +-2.5 t/s, pp16384 467.1 +-1.0 t/s (r=5) tg128 10.81 +-0.02 t/s, unchanged Decode and spec-verify batches never select the large tile (n<=64), and greedy outputs are byte-identical. Tile-area probes beyond 128x256 (BM=256, 1024 threads, TM/TN/WMITER reshapings, wave32) were neutral; the coopmat quant family (iq4_xs dequant path) measurably dislikes the doubled tile and stays stock. Gated to RDNA3 + integrated GPU. GGML_VK_MMQ_INT_K / GGML_VK_MMQ_INT / GGML_VK_MMQ_CM env probes are added for further tuning; wg_denoms are kept in sync with BM/BN so dispatch and split-k see the real tile.
aic0d3r
force-pushed
the
gfx1151-matmul
branch
from
August 22, 2026 17:22
12301f9 to
0c116e7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Dense-model prefill on gfx1151 is int-mmq-K-quant bound, and the stock large tile underfills the 40 CUs at prefill shapes: it re-reads the A weights once per 128-token slab of N. Doubling only BN and the workgroup size (128x256 @ 512 threads, per-thread TM/TN/WMITER unchanged) halves the A re-reads per token. No shader changes;
wg_denomskept in sync. Gated to RDNA3 + iGPU.Stacked on strix-halo-vulkan @ 2586f6e, Qwen3.8-27B UD-Q5_K_XL-v2, bundled RADV,
-fa 1 -b/-ub 2048:Deep context at the serving config (
-b/-ub 4096, same model/driver; stock arm tile-forced back via the env on the same binary, pp65536 stock is from the v0.6.5 payload — same engine family, corroborated by pp32768: 241.3 here vs 251.5 there):* pp16384 stock was measured on mainline (
b21e4de74), not on the fork — the stock tile value is identical in both trees, but the fork's long-ctx prefill fixes mean fork-stock is likely somewhat higher, so treat 2.16x as an upper bound. Patched numbers are fork, r=5. The pp4096 A/B is same-binary interleaved (3 rounds, tile forced back viaGGML_VK_MMQ_INT_K); pp512 stock is r=3 with the same env forcing.Correctness and gates:
GGML_VK_MMQ_INT_K(int K-quants),GGML_VK_MMQ_INT(int non-K, mirrors K when unset),GGML_VK_MMQ_CM(coopmat quant family).Also upstreamed to ggml-org: ggml-org#27554 (issue ggml-org#27553) — stock and patched both lower on mainline at every length (stock 250 vs 289, patched 437 vs 527 pp4096; patched pp16384 333 vs 467), so the fork's long-ctx prefill work is clearly stacking with this.
One datapoint for the
3ca6f21f2RADV pad-2 gate, in case it's useful: on system Mesa 26.2.0 (Arch,mesa 3:26.2.0-1), dense pp4096 with pad 2 measures 222 vs 284 withGGML_VK_SHMEM_PAD=4(−22%). Your table has 26.2.1 winning on the MoE pp512 shape, but 26.2.0 on dense leaks through the>= 25.3.0version gate. The start-script workaround pins the bundled RADV, so nothing urgent.