Skip to content

Optimizations for q40, q80, q5k, fix Cosmos VLM prefill - #92

Open
liangliangchang wants to merge 7 commits into
gfx11from
lichang.mmq-opt-cont
Open

Optimizations for q40, q80, q5k, fix Cosmos VLM prefill#92
liangliangchang wants to merge 7 commits into
gfx11from
lichang.mmq-opt-cont

Conversation

@liangliangchang

@liangliangchang liangliangchang commented Aug 25, 2026 •
edited
Loading

Copy link
Copy Markdown
  1. Fixed ntx size for Q4_0 and Q8_0, which affect several VLMs.
  2. Optimized q40, q80, q5k mmq kernels with Nibble-split, Batch J=128 WMMAs, X prefetch, Raise MMQ N cutoff

Highlights: prefill gains on the shapes the MMQ work targets

row quant base prefill tok/s opt prefill tok/s prefill %
Gemma-4-E2B-IT_Q4_0_GGUF_VLM † Q4_0 978.0 1878.2 +92.05
Gemma-4-E2B-IT_Q4_0_GGUF_VLM_multi-image-reasoning † Q4_0 492.2 909.6 +84.80
Gemma-4-E2B-IT_Q4_0_GGUF_VLM_image-caption-224 † Q4_0 472.1 793.6 +68.11
Gemma-4-E2B-IT_Q4_0_GGUF_VLM_vqa-448 † Q4_0 489.3 821.9 +67.95
Cosmos-Reason2-8B_Q8_0_GGUF_VLM_prithivMLmods † Q8_0 589.1 863.5 +46.59
Gemma-4-26B-A4B-IT_VLM † mixed 701.9 834.8 +18.94
Gemma-4-E4B-IT_Q4_0_GGUF_VLM_image-caption-224 Q4_0 300.0 326.0 +8.67
Qwen3.5-4B_Q4_K_M_GGUF_mtp_prose Q4_K_M 290.1 298.9 +3.04

Q4_0: Latency and throughput

MUL_MAT shape (m × n × k) gfx11 (µs) q4-0-opt (µs) gfx11 TFLOPS q4-0-opt TFLOPS Δ TFLOPS
1024 × 128 × 4096 79.64 69.60 13.52 15.43 +14.1%
12288 × 32 × 4096 248.93 235.95 12.94 13.66 +5.5%
4096 × 32 × 12288 297.90 289.49 10.83 11.12 +2.7%
64 × 128 × 4096 66.91 65.53 1.00 1.02 +2.0%
4096 × 4 × 4096 29.47 29.24 4.55 4.59 +0.8%
4096 × 32 × 4096 103.48 103.62 10.38 10.37 −0.1%
4096 × 512 × 14336 2455.40 2457.42 24.49 24.47 −0.1%
4096 × 8 × 14336 165.00 165.21 5.70 5.69 −0.2%
4096 × 128 × 12288 616.05 619.37 20.92 20.81 −0.5%
4096 × 128 × 4096 198.57 199.85 21.63 21.49 −0.6%
8192 × 128 × 4096 340.09 342.86 25.26 25.05 −0.8%
12288 × 128 × 4096 498.37 502.49 25.86 25.64 −0.8%
32 × 128 × 4096 68.96 71.56 0.49 0.47 −3.4%

Q8_0 Latency and throughput

MUL_MAT shape (m × n × k) gfx11 (µs) q8-0-opt (µs) gfx11 TFLOPS q8-0-opt TFLOPS Δ TFLOPS
8 × 128 × 4096 94.39 61.67 0.089 0.136 +52.9%
16 × 128 × 4096 92.45 62.75 0.181 0.267 +47.3%
32 × 128 × 4096 93.27 62.75 0.360 0.535 +48.6%
64 × 128 × 4096 69.48 61.22 0.97 1.10 +13.2%
1024 × 128 × 4096 69.92 67.06 15.36 16.01 +4.2%
12288 × 128 × 4096 512.75 497.07 25.13 25.92 +3.2%
4096 × 512 × 14336 2462.20 2405.64 24.42 25.00 +2.4%
8192 × 128 × 4096 355.01 353.58 24.20 24.30 +0.4%
4096 × 4 × 4096 35.75 35.60 3.76 3.77 +0.4%
4096 × 128 × 4096 191.53 190.98 22.43 22.49 +0.3%
4096 × 8 × 14336 303.66 304.52 3.09 3.09 −0.2%

Q5_k Latency and throughput

MUL_MAT shape (m × n × k) gfx11 (µs) final (µs) paired speedup
8192 × 128 × 2560 245.00 224.60 +9.03%
8192 × 512 × 2560 914.88 844.89 +8.16%
2560 × 128 × 4096 123.31 119.88 +2.85%
1024 × 128 × 4096 68.85 68.12 +1.06%
4096 × 128 × 4096 212.24 205.76 +3.14%

Model performance

row base decode opt decode decode % base prefill opt prefill prefill % status
Cosmos-Reason2-8B_Q8_0_GGUF_VLM_prithivMLmods † 25.93 25.93 +0.00 589.10 863.54 +46.59 PASS
GLM-4.7-Flash_Q4_K_M_GGUF_128 63.09 63.12 +0.05 1057.13 1051.74 -0.51 PASS
Gemma-2B_Q4_K_M_GGUF_128 119.41 119.50 +0.07 4043.15 4077.15 +0.84 PASS
Gemma-2B_Q4_K_M_GGUF_8000 110.54 110.55 +0.01 2373.57 2375.80 +0.09 PASS
Gemma-3-12B-IT_Q4_K_M_GGUF_4096 25.11 25.12 +0.01 839.30 838.72 -0.07 PASS
Gemma-3-12B-IT_Q4_K_M_GGUF_VLM 27.38 27.40 +0.06 275.73 276.18 +0.17 PASS
Gemma-3-4B-IT_VLM 70.40 70.39 -0.02 671.35 669.40 -0.29 PASS
Gemma-3-4B-IT_VLM_document-qa-long-context 71.90 71.93 +0.05 2053.14 2055.09 +0.09 PASS
Gemma-3-4B-IT_VLM_image-caption-224 71.04 71.00 -0.06 299.55 300.04 +0.16 PASS
Gemma-3-4B-IT_VLM_multi-image-reasoning 71.16 71.07 -0.13 288.01 288.75 +0.26 PASS
Gemma-3-4B-IT_VLM_ocr-1024 72.08 72.06 -0.02 299.79 298.72 -0.36 PASS
Gemma-3-4B-IT_VLM_text-short-smoke 68.46 70.24 +2.60 392.06 401.60 +2.43 PASS
Gemma-3-4B-IT_VLM_vqa-448 69.46 69.47 +0.00 293.25 293.10 -0.05 PASS
Gemma-4-12B-it_Q4_K_M_GGUF_128 28.61 28.60 -0.05 894.05 894.08 +0.00 PASS
Gemma-4-26B-A4B-IT_VLM † 45.95 45.95 +0.00 701.89 834.81 +18.94 PASS
Gemma-4-31B-IT_VLM 10.66 10.67 +0.07 243.01 244.01 +0.41 PASS
Gemma-4-31B-IT_VLM_OpenNav 10.74 10.75 +0.11 230.82 230.71 -0.05 PASS
Gemma-4-E2B-IT_BF16_GGUF_128 43.36 43.37 +0.01 3122.94 3144.47 +0.69 PASS
Gemma-4-E2B-IT_Q4_0_GGUF_VLM 100.93 100.91 -0.01 975.61 974.36 -0.13 PASS
Gemma-4-E2B-IT_Q4_0_GGUF_VLM_document-qa-long-context 95.81 96.14 +0.35 3222.82 3229.22 +0.20 PASS
Gemma-4-E2B-IT_Q4_0_GGUF_VLM_image-caption-224 92.11 92.80 +0.75 470.68 521.45 +10.79 PASS
Gemma-4-E2B-IT_Q4_0_GGUF_VLM_multi-image-reasoning 98.29 98.39 +0.10 490.24 489.72 -0.11 PASS
Gemma-4-E2B-IT_Q4_0_GGUF_VLM_ocr-1024 100.37 100.29 -0.08 1014.79 1011.64 -0.31 PASS
Gemma-4-E2B-IT_Q4_0_GGUF_VLM_text-short-smoke 76.05 82.05 +7.89 547.43 554.27 +1.25 PASS
Gemma-4-E2B-IT_Q4_0_GGUF_VLM_vqa-448 84.35 84.60 +0.30 488.40 486.61 -0.37 PASS
Gemma-4-E2B-IT_Q4_K_M_GGUF_128 105.74 107.75 +1.91 3345.37 3343.93 -0.04 PASS
Gemma-4-E2B-IT_Q4_K_M_GGUF_3968 101.51 101.44 -0.06 2852.72 2871.51 +0.66 PASS
Gemma-4-E4B-IT_Q4_0_GGUF_VLM 56.56 56.55 -0.01 595.32 594.58 -0.12 PASS
Gemma-4-E4B-IT_Q4_0_GGUF_VLM_document-qa-long-context 55.79 55.83 +0.06 1890.10 1891.77 +0.09 PASS
Gemma-4-E4B-IT_Q4_0_GGUF_VLM_image-caption-224 54.83 54.86 +0.05 299.96 325.98 +8.67 PASS
Gemma-4-E4B-IT_Q4_0_GGUF_VLM_multi-image-reasoning 56.51 56.48 -0.05 322.82 320.78 -0.63 PASS
Gemma-4-E4B-IT_Q4_0_GGUF_VLM_ocr-1024 57.01 57.11 +0.18 784.52 780.89 -0.46 PASS
Gemma-4-E4B-IT_Q4_0_GGUF_VLM_text-short-smoke 46.88 49.84 +6.31 344.66 346.05 +0.40 PASS
Gemma-4-E4B-IT_Q4_0_GGUF_VLM_vqa-448 51.87 51.77 -0.17 313.58 313.17 -0.13 PASS
Janus-Pro-1B_VLM 150.62 150.53 -0.06 4504.43 4500.51 -0.09 PASS
Janus-Pro-7B_VLM 41.03 41.06 +0.07 1422.28 1421.13 -0.08 PASS
Llama-2-7B_Q4_K_M_GGUF_128 49.52 49.66 +0.27 1344.48 1359.47 +1.11 PASS
Llama-2-7B_Q4_K_M_GGUF_1920 35.78 35.79 +0.02 1176.54 1183.32 +0.58 PASS
Llama-3.1-8B-Instruct_Q4_K_M_GGUF_128 44.39 44.40 +0.02 1301.83 1311.66 +0.76 PASS
MiniCPM-V-2_6_VLM 46.27 46.29 +0.04 589.60 589.65 +0.01 PASS
MiniCPM-o-2_6_VLM 46.26 46.26 +0.01 596.17 597.17 +0.17 PASS
Qwen2.5-0.5B-Instruct_Q4_K_M_GGUF_128 347.42 278.70 -19.78 10073.25 10047.35 -0.26 PASS
Qwen2.5-0.5B-Instruct_Q4_K_M_GGUF_3968 315.82 315.09 -0.23 10128.60 10143.45 +0.15 PASS
Qwen2.5-3B-Instruct_Q4_K_M_GGUF_128 85.16 96.14 +12.89 2886.11 2904.97 +0.65 PASS
Qwen2.5-3B-Instruct_Q4_K_M_GGUF_3968 90.10 90.06 -0.05 2724.53 2730.35 +0.21 PASS
Qwen2.5-7B-Instruct_Q4_K_M_GGUF_128 47.68 47.83 +0.30 1519.22 1534.45 +1.00 PASS
Qwen2.5-7B-Instruct_Q4_K_M_GGUF_3968 44.51 44.51 +0.00 1378.53 1381.71 +0.23 PASS
Qwen2.5-VL-3B-Instruct_VLM 90.97 90.98 +0.01 706.46 707.34 +0.13 PASS
Qwen2.5-VL-7B-Instruct_VLM 45.56 45.56 +0.02 578.14 579.04 +0.16 PASS
Qwen3-1.7B_Q4_K_M_GGUF_128 157.95 158.11 +0.10 4753.71 4825.35 +1.51 PASS
Qwen3-1.7B_Q4_K_M_GGUF_3968 117.78 117.76 -0.02 3751.86 3772.61 +0.55 PASS
Qwen3-30B-A3B-Instruct-2507_Q4_K_M_GGUF_128 80.56 82.20 +2.03 1299.14 1279.64 -1.50 PASS
Qwen3-4B_Q4_K_M_GGUF_128 77.46 77.57 +0.14 2217.25 2237.28 +0.90 PASS
Qwen3-4B_Q4_K_M_GGUF_3968 63.88 63.85 -0.05 1761.36 1764.74 +0.19 PASS
Qwen3-8B_Q4_K_M_GGUF_128 43.55 43.53 -0.06 1279.47 1300.98 +1.68 PASS
Qwen3-8B_Q4_K_M_GGUF_3968 38.83 38.83 +0.02 1178.57 1178.12 -0.04 PASS
Qwen3-Omni-30B-A3B-Instruct_VLM 75.77 75.67 -0.14 1001.00 999.96 -0.10 PASS
Qwen3-VL-4B-Instruct_VLM 70.24 70.19 -0.08 1251.96 1253.68 +0.14 PASS
Qwen3.5-35B-A3B_Q4_K_M_GGUF_128 60.06 60.06 -0.01 1306.28 1334.22 +2.14 PASS
Qwen3.5-35B-A3B_VLM 58.75 58.71 -0.07 913.97 914.06 +0.01 PASS
Qwen3.5-4B_Q4_0_GGUF_baseline_code 65.62 65.65 +0.04 473.67 473.43 -0.05 PASS
Qwen3.5-4B_Q4_0_GGUF_baseline_list 65.16 65.19 +0.05 398.70 392.04 -1.67 PASS
Qwen3.5-4B_Q4_0_GGUF_baseline_prose 65.53 65.55 +0.03 335.24 334.97 -0.08 PASS
Qwen3.5-4B_Q4_0_GGUF_mtp_code 123.21 123.13 -0.06 400.40 394.11 -1.57 PASS
Qwen3.5-4B_Q4_0_GGUF_mtp_list 127.56 127.72 +0.13 329.53 327.74 -0.54 PASS
Qwen3.5-4B_Q4_0_GGUF_mtp_prose 86.25 86.43 +0.21 280.33 277.13 -1.14 PASS
Qwen3.5-4B_Q4_K_M_GGUF_128 63.56 63.53 -0.05 2007.49 2040.28 +1.63 PASS
Qwen3.5-4B_Q4_K_M_GGUF_baseline_code 60.21 60.30 +0.15 500.63 512.16 +2.30 PASS
Qwen3.5-4B_Q4_K_M_GGUF_baseline_list 59.78 59.86 +0.14 420.30 415.86 -1.06 PASS
Qwen3.5-4B_Q4_K_M_GGUF_baseline_prose 60.14 60.22 +0.13 349.87 355.82 +1.70 PASS
Qwen3.5-4B_Q4_K_M_GGUF_mtp_code 106.06 106.30 +0.23 418.78 424.30 +1.32 PASS
Qwen3.5-4B_Q4_K_M_GGUF_mtp_list 108.43 109.40 +0.90 344.44 348.96 +1.31 PASS
Qwen3.5-4B_Q4_K_M_GGUF_mtp_prose 69.11 69.30 +0.28 290.11 298.93 +3.04 PASS
Qwen3.5-9B_Q4_0_GGUF_baseline_code 39.61 39.63 +0.03 322.18 325.65 +1.08 PASS
Qwen3.5-9B_Q4_0_GGUF_baseline_list 39.47 39.46 -0.02 263.33 265.83 +0.95 PASS
Qwen3.5-9B_Q4_0_GGUF_baseline_prose 39.60 39.62 +0.05 226.88 228.13 +0.55 PASS
Qwen3.5-9B_Q4_0_GGUF_mtp_code 81.59 81.56 -0.04 288.74 286.55 -0.76 PASS
Qwen3.5-9B_Q4_0_GGUF_mtp_list 83.77 83.84 +0.09 233.27 237.25 +1.71 PASS
Qwen3.5-9B_Q4_0_GGUF_mtp_prose 57.48 57.43 -0.07 199.80 201.26 +0.73 PASS
Qwen3.5-9B_Q4_K_M_GGUF_128 37.72 37.69 -0.07 1202.77 1233.84 +2.58 PASS
Qwen3.5-9B_Q4_K_M_GGUF_baseline_code 37.13 37.12 -0.04 365.38 363.38 -0.55 PASS
Qwen3.5-9B_Q4_K_M_GGUF_baseline_list 36.94 36.98 +0.10 297.95 299.26 +0.44 PASS
Qwen3.5-9B_Q4_K_M_GGUF_baseline_prose 37.10 37.12 +0.05 255.84 255.50 -0.13 PASS
Qwen3.5-9B_Q4_K_M_GGUF_mtp_code 68.93 68.85 -0.11 318.32 320.05 +0.54 PASS
Qwen3.5-9B_Q4_K_M_GGUF_mtp_list 73.08 73.36 +0.38 257.84 261.18 +1.29 PASS
Qwen3.5-9B_Q4_K_M_GGUF_mtp_prose 46.30 46.27 -0.05 220.74 223.32 +1.17 PASS
Qwen3.6-27B_Q4_0_GGUF_baseline_code 13.35 13.35 -0.02 112.19 111.52 -0.59 PASS
Qwen3.6-27B_Q4_0_GGUF_baseline_list 13.32 13.32 -0.03 91.44 90.15 -1.41 PASS
Qwen3.6-27B_Q4_0_GGUF_baseline_prose 13.34 13.36 +0.10 77.78 77.24 -0.69 PASS
Qwen3.6-27B_Q4_0_GGUF_mtp_code 34.41 34.43 +0.06 98.48 100.12 +1.67 PASS
Qwen3.6-27B_Q4_0_GGUF_mtp_list 35.40 35.41 +0.01 80.71 81.32 +0.75 PASS
Qwen3.6-27B_Q4_0_GGUF_mtp_prose 27.97 27.96 -0.04 69.12 69.31 +0.27 PASS
Qwen3.6-27B_Q4_K_M_GGUF_4096 12.31 12.31 +0.00 380.51 383.95 +0.90 PASS
Qwen3.6-27B_Q4_K_M_GGUF_baseline_code 12.44 12.45 +0.06 125.76 127.77 +1.60 PASS
Qwen3.6-27B_Q4_K_M_GGUF_baseline_list 12.39 12.40 +0.05 101.56 104.10 +2.50 PASS
Qwen3.6-27B_Q4_K_M_GGUF_baseline_prose 12.33 12.45 +0.99 88.44 89.63 +1.34 PASS
Qwen3.6-27B_Q4_K_M_GGUF_mtp_code 28.92 28.92 -0.01 111.00 110.64 -0.32 PASS
Qwen3.6-27B_Q4_K_M_GGUF_mtp_list 29.02 29.04 +0.05 90.65 90.50 -0.17 PASS
Qwen3.6-27B_Q4_K_M_GGUF_mtp_prose 19.55 19.55 +0.01 77.26 77.49 +0.30 PASS
Qwen3.6-35B-A3B_Q4_K_M_GGUF_128 60.80 60.81 +0.01 1307.37 1310.59 +0.25 PASS
Qwen3.6-35B-A3B_Q4_K_M_GGUF_4096 59.43 59.36 -0.12 1618.74 1618.95 +0.01 PASS
Qwen3.6-35B-A3B_Q4_K_M_GGUF_baseline_code 59.04 59.06 +0.03 299.54 296.61 -0.98 PASS
Qwen3.6-35B-A3B_Q4_K_M_GGUF_baseline_list 57.95 57.94 -0.01 249.96 255.58 +2.25 PASS
Qwen3.6-35B-A3B_Q4_K_M_GGUF_baseline_prose 58.90 58.91 +0.01 216.03 214.86 -0.54 PASS
Qwen3.6-35B-A3B_Q4_K_M_GGUF_mtp_code 96.47 96.52 +0.05 255.57 258.13 +1.00 PASS
Qwen3.6-35B-A3B_Q4_K_M_GGUF_mtp_list 91.00 90.65 -0.38 215.43 220.39 +2.30 PASS
Qwen3.6-35B-A3B_Q4_K_M_GGUF_mtp_prose 69.40 69.42 +0.02 185.25 185.34 +0.05 PASS
Qwen3.6-35B-A3B_VLM_large-context 54.51 54.53 +0.03 1502.56 1510.21 +0.51 PASS
SmolLM2-1.7B-Instruct_F16_GGUF_128 53.21 53.25 +0.07 4017.24 4006.28 -0.27 PASS
SmolLM2-1.7B-Instruct_F16_GGUF_8000 25.42 25.43 +0.01 1268.14 1268.46 +0.03 PASS
SmolLM2-1.7B-Instruct_Q4_K_M_GGUF_128 161.08 161.09 +0.00 4193.60 4215.55 +0.52 PASS
SmolLM2-1.7B-Instruct_Q4_K_M_GGUF_8000 37.31 37.31 +0.01 1224.73 1228.51 +0.31 PASS

liangliangchang and others added 7 commits August 25, 2026 17:54
@liangliangchang
HIP: batch RDNA3.5 Q4_0 J128 WMMA and prefetch X tiles
4e564f3 
Apply the Q4_K J=128 WMMA split and next-K X prefetch to Q4_0 MMQ.
@liangliangchang @cursoragent
HIP: batch RDNA3.5 Q8_0 J128 WMMA and prefetch X tiles
4760c1a 
Co-authored-by: Cursor <cursoragent@cursor.com>
@liangliangchang @cursoragent
HIP: batch RDNA3.5 Q5_K J128 WMMA like Q4_K
7050612 
Q5_K unpacks to the same IU8 SRAM as Q4_K, so reuse the J=128 ntx=1
wmma_low / convert / wmma_high schedule and the Q4_K J policy.

Co-authored-by: Cursor <cursoragent@cursor.com>
@liangliangchang @cursoragent
HIP: pipeline RDNA3.5 Q5_K tile loads
8a6872b 
Prefetch Q5_K low- and high-bit data during WMMA while distributing high bits across lanes to keep J128 spill-free.

Co-authored-by: Cursor <cursoragent@cursor.com>
@liangliangchang @cursoragent
HIP: limit RDNA3.5 Q8_0 X-tile prefetch to J=128
ad9d427 
The prefetch pipeline was gated on J == 96 || J == 128, but the batched WMMA
vec_dot it feeds is gated on J == 128 only, and rows_per_warp() returns 16
(ntx=1) only at J == 128 -- at J == 96 it returns 32 (ntx=2). So J == 96 paired
the prefetch with the generic two-minitile kernel. Q8_0 is also the only type
caching 2*(I/nwarps) = 32 ints where the others cache 16, on top of two A tiles
and 48 accumulators, so that combination spills.

mmq_rdna35_tuned_J maps Q8_0 J_occupancy == 64 to 96 to avoid the J == 64
pathology, which made every 64-wide batch land on the broken width. MoE Q8_0
(J > 32 -> 96) had the same exposure.

Cosmos-Reason2-8B Q8_0 prefill on gfx1151, llama-bench -p 512 -r 3 against
rocm/gfx11 03d2068:

  ubatch  64:  111.4 -> 178.7 t/s  (was -37.7% vs gfx11, now -0.1%)
  ubatch  96:    n/a -> 267.6 t/s  (now -0.4% vs gfx11)
  ubatch 128: 1426.8 -> 1430.9 t/s (+3.0% vs gfx11, unchanged)
  ubatch 512: 1581.5 -> 1581.9 t/s (+2.8% vs gfx11, unchanged)

The J == 128 win is retained in full. test-backend-ops MUL_MAT q8_0: 64/64.

Co-authored-by: Cursor <cursoragent@cursor.com>
@liangliangchang @cursoragent
HIP: keep RDNA3.5 block quants at ntx=1 for J=64 and J=96
2a6e364 
Q4_0 and Q8_0 only have a batched WMMA vec_dot at J=128. At the other two widths
where rows_per_warp() would pick 32 (J=64 and J=96, the only J>=64 multiples of
32) they fall back to the generic two-minitile schedule, which is several times
slower on gfx115x. J=72/80/88/104/112/120 are unaffected because they are not
multiples of 32 and already run ntx=1.

This is a pre-existing cliff, not a regression: gfx11 shows it too. Q4_K is not
affected, since its generic path handles ntx=2 well.

llama-bench -p 512 -r 3 on gfx1151, versus rocm/gfx11 03d2068:

  Cosmos-Reason2-8B Q8_0   ubatch  64:  178.9 ->  883.1 t/s  (+394%)
                           ubatch  96:  268.8 -> 1151.4 t/s  (+328%)
  Qwen3-14B Q4_0           ubatch  64:  153.6 ->  708.7 t/s  (+361%)
                           ubatch  96:  156.8 ->  725.8 t/s  (+363%)

Unaffected widths are unchanged (Q8_0 ubatch 32: 534.5 -> 533.1, ubatch 128:
1389.7 -> 1416.1). test-backend-ops MUL_MAT q4_0+q8_0: 134/134.

Co-authored-by: Cursor <cursoragent@cursor.com>
@liangliangchang @cursoragent
HIP: drop the RDNA3.5 q8_0 J=64 -> J=96 widening
c0a6c7c 
The widening existed because J=64 was several times slower for q8_0, but that was
the generic ntx=2 schedule which rows_per_warp() no longer selects for block
quants. With J=64 healthy the widening only pads 64 valid columns into a 96-wide
tile.

Cosmos-Reason2-8B Q8_0, llama-bench -p 512 -r 3 on gfx1151:

  ubatch  64 (J 96 -> 64): 883.1 -> 1118.5 t/s  (+26.7%)
  ubatch  96 (J 96, same): 1151.4 -> 1147.2 t/s (-0.4%)
  ubatch 128 (J 128, same): 1416.1 -> 1398.1 t/s (-1.3%)

MoE keeps its own widening, which is untested here and unchanged.
test-backend-ops MUL_MAT: 1484/1484.

Co-authored-by: Cursor <cursoragent@cursor.com>
@liangliangchang liangliangchang changed the title Some more optimizations for q40, q80, q6k, q5k Optimizations for q40, q80, q5k, fix Cosmos VLM prefill Sep 1, 2026
@liangliangchang
liangliangchang marked this pull request as ready for review September 1, 2026 19:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jimw567 jimw567 Awaiting requested review from jimw567
@Annieren Annieren Awaiting requested review from Annieren
@mgehre-amd mgehre-amd Awaiting requested review from mgehre-amd

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@liangliangchang