Skip to content

vulkan: double the K-quant int-mmq large tile on RDNA3 iGPUs - #27554

Open
aic0d3r wants to merge 1 commit into
ggml-org:masterfrom
aic0d3r:gfx1151-mmq-tile
Open

vulkan: double the K-quant int-mmq large tile on RDNA3 iGPUs#27554
aic0d3r wants to merge 1 commit into
ggml-org:masterfrom
aic0d3r:gfx1151-mmq-tile

Conversation

@aic0d3r

@aic0d3r aic0d3r commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

Implements #27553.

The integer-dot mmq path picks its large tile when both matmul dims exceed 64, which is every big GEMM in a dense-model prefill. On RDNA3 iGPUs (gfx1151-class, e.g. Radeon 8060S) the stock 128x128 tile at 256 threads underfills the shader cores at prefill shapes and re-reads the A weights once per 128-token slab; doubling only BN and the thread count keeps the per-thread register tiling identical while halving the A re-reads per token. No shader changes; wg_denoms kept in sync so dispatch and split-k see the real tile.

Qwen3.8-27B UD-Q5_K_XL-v2, RADV, -fa 1 -b/-ub 2048, same-binary stash A/B, r=5:

test stock 128x256@512 gain
pp512 294.5 ±6.4 517.0 ±3.4 1.76x
pp4096 249.6 ±0.2 436.7 ±0.8 1.75x
pp16384 216.1 ±0.4 333.2 ±1.7 1.54x
tg128 10.76 10.82 unchanged

Also measured (r=3, pp4096): Q4_K_XL 465.1 ±0.8, Q6_K_XL 411.8 ±1.3 with the new tile.

Correctness and scope:

  • Greedy outputs byte-identical (temp 0, diffed, -n 120).
  • Decode and speculative-verify batches (n ≤ 64) select the small/medium tile and are structurally unaffected — verified with an MTP/DFlash2 spec-decode serving config: 28.7 t/s average decode at 64% draft acceptance, no regression vs baseline.
  • Gated to AMD RDNA3 && device->uma; discrete GPUs, other vendors, and all mul_mat_id (MoE) tile families untouched.
  • Neutral probes on this hardware: BM=256, 1024 threads, TM/TN/WMITER reshapings, wave32. The coopmat dequant family (iq4_xs path) measurably dislikes the doubled tile and stays stock.

@aic0d3r
aic0d3r requested a review from a team as a code owner August 22, 2026 17:05
@github-actions github-actions Bot added Vulkan Issues specific to the Vulkan backend ggml changes relating to the ggml tensor library for machine learning labels Aug 22, 2026
The integer-dot mmq path picks its large tile when both matmul dims
exceed 64, which is every big GEMM in a dense-model prefill. On RDNA3
iGPUs (gfx1151-class, e.g. Radeon 8060S) the stock 128x128 tile at 256
threads underfills the shader cores at prefill shapes and re-reads the
A weights once per 128-token slab; doubling only BN and the thread
count keeps the per-thread register tiling identical while halving the
A re-reads per token.

Qwen3.8-27B UD-Q5_K_XL-v2, RADV, -fa 1 -b/-ub 2048, r=5, same binary
stash A/B:

  pp512    294.5 +-6.4  ->  517.0 +-3.4   (1.76x)
  pp4096   249.6 +-0.2  ->  436.7 +-0.8   (1.75x)
  pp16384  216.1 +-0.4  ->  333.2 +-1.7   (1.54x)
  tg128    10.76        ->  10.82         (unchanged)

Decode and verify batches (n <= 64) select the small or medium tile
and are unaffected; greedy outputs are byte-identical. wg_denoms are
kept in sync with the tile so dispatch and split-k see the real shape.
Gated to AMD RDNA3 + integrated GPU via device->uma.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ggml changes relating to the ggml tensor library for machine learning Vulkan Issues specific to the Vulkan backend

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant