Skip to content

HIP: Expand Q5_K and Q6_K tile widths for RDNA2 - #27558

Open
draetheus wants to merge 2 commits into
ggml-org:masterfrom
draetheus:expand-q5k-q6k-tile-widths
Open

HIP: Expand Q5_K and Q6_K tile widths for RDNA2#27558
draetheus wants to merge 2 commits into
ggml-org:masterfrom
draetheus:expand-q5k-q6k-tile-widths

Conversation

@draetheus

@draetheus draetheus commented Aug 22, 2026

Copy link
Copy Markdown

Overview

I recently performed a tile sweep of the RDNA2 config to see if there were any easy performance wins. Most of the config is optimal as is, however Q6_K and Q5_K see a noticeable performance increase by allowing J=128. The intermediate values do not seem to have an impact in benchmarks but were added for consistency.

Additional information

Test results from my Radeon Pro V620 on ROCm 7.14. I'm seeing up to 11.5% increase for Q6_K and up to 7.5% for Q5_K for classic quants. In my test of Qwen 3.8 27B, which is a more realistic mixed quant, I still see up to 8% increase. All test ops pass. Will post perplexity if requested.

Note that the small classic quants experience regression at pp128, while the larger mixed qwen model does not. I think this is an acceptable trade off but subject to debate and possible fixes.

Master

llama 8B Q5_K
model size params backend ngl test t/s
llama 8B Q5_K - Small 5.21 GiB 8.03 B ROCm -1 pp128 1217.14 ± 58.55
llama 8B Q5_K - Small 5.21 GiB 8.03 B ROCm -1 pp256 1394.38 ± 18.03
llama 8B Q5_K - Small 5.21 GiB 8.03 B ROCm -1 pp512 1461.10 ± 10.38
llama 8B Q5_K - Small 5.21 GiB 8.03 B ROCm -1 pp1024 1441.81 ± 1.81
llama 8B Q5_K - Small 5.21 GiB 8.03 B ROCm -1 pp2048 1399.76 ± 1.66
llama 8B Q5_K - Small 5.21 GiB 8.03 B ROCm -1 pp128 @ d8192 892.67 ± 19.68
llama 8B Q5_K - Small 5.21 GiB 8.03 B ROCm -1 pp256 @ d8192 981.11 ± 11.19
llama 8B Q5_K - Small 5.21 GiB 8.03 B ROCm -1 pp512 @ d8192 1014.52 ± 6.15
llama 8B Q5_K - Small 5.21 GiB 8.03 B ROCm -1 pp1024 @ d8192 1003.50 ± 0.28
llama 8B Q5_K - Small 5.21 GiB 8.03 B ROCm -1 pp2048 @ d8192 982.25 ± 0.14
llama 8B Q6_K
model size params backend ngl test t/s
llama 8B Q6_K 6.14 GiB 8.03 B ROCm -1 pp128 1041.73 ± 42.84
llama 8B Q6_K 6.14 GiB 8.03 B ROCm -1 pp256 1185.40 ± 13.33
llama 8B Q6_K 6.14 GiB 8.03 B ROCm -1 pp512 1244.74 ± 7.40
llama 8B Q6_K 6.14 GiB 8.03 B ROCm -1 pp1024 1232.88 ± 0.70
llama 8B Q6_K 6.14 GiB 8.03 B ROCm -1 pp2048 1204.22 ± 0.54
llama 8B Q6_K 6.14 GiB 8.03 B ROCm -1 pp128 @ d8192 794.21 ± 15.83
llama 8B Q6_K 6.14 GiB 8.03 B ROCm -1 pp256 @ d8192 872.66 ± 8.86
llama 8B Q6_K 6.14 GiB 8.03 B ROCm -1 pp512 @ d8192 906.13 ± 4.86
llama 8B Q6_K 6.14 GiB 8.03 B ROCm -1 pp1024 @ d8192 898.25 ± 0.25
llama 8B Q6_K 6.14 GiB 8.03 B ROCm -1 pp2048 @ d8192 882.34 ± 0.24
qwen35 27B Q6_K
model size params backend ngl test t/s
qwen35 27B Q6_K 21.49 GiB 27.32 B ROCm -1 pp128 316.77 ± 16.41
qwen35 27B Q6_K 21.49 GiB 27.32 B ROCm -1 pp256 361.67 ± 8.22
qwen35 27B Q6_K 21.49 GiB 27.32 B ROCm -1 pp512 384.91 ± 4.38
qwen35 27B Q6_K 21.49 GiB 27.32 B ROCm -1 pp1024 383.11 ± 0.58
qwen35 27B Q6_K 21.49 GiB 27.32 B ROCm -1 pp2048 379.06 ± 0.25
qwen35 27B Q6_K 21.49 GiB 27.32 B ROCm -1 pp128 @ d8192 294.31 ± 10.80
qwen35 27B Q6_K 21.49 GiB 27.32 B ROCm -1 pp256 @ d8192 330.28 ± 6.64
qwen35 27B Q6_K 21.49 GiB 27.32 B ROCm -1 pp512 @ d8192 350.67 ± 3.75
qwen35 27B Q6_K 21.49 GiB 27.32 B ROCm -1 pp1024 @ d8192 350.74 ± 0.06
qwen35 27B Q6_K 21.49 GiB 27.32 B ROCm -1 pp2048 @ d8192 348.68 ± 0.04

Branch

llama 8B Q5_K
model size params backend ngl test t/s
llama 8B Q5_K - Small 5.21 GiB 8.03 B ROCm -1 pp128 1072.40 ± 49.93
llama 8B Q5_K - Small 5.21 GiB 8.03 B ROCm -1 pp256 1451.47 ± 20.40
llama 8B Q5_K - Small 5.21 GiB 8.03 B ROCm -1 pp512 1572.89 ± 11.40
llama 8B Q5_K - Small 5.21 GiB 8.03 B ROCm -1 pp1024 1548.65 ± 1.66
llama 8B Q5_K - Small 5.21 GiB 8.03 B ROCm -1 pp2048 1498.85 ± 1.44
llama 8B Q5_K - Small 5.21 GiB 8.03 B ROCm -1 pp128 @ d8192 802.70 ± 18.99
llama 8B Q5_K - Small 5.21 GiB 8.03 B ROCm -1 pp256 @ d8192 1010.06 ± 11.85
llama 8B Q5_K - Small 5.21 GiB 8.03 B ROCm -1 pp512 @ d8192 1067.26 ± 7.07
llama 8B Q5_K - Small 5.21 GiB 8.03 B ROCm -1 pp1024 @ d8192 1053.09 ± 0.20
llama 8B Q5_K - Small 5.21 GiB 8.03 B ROCm -1 pp2048 @ d8192 1029.58 ± 0.63
llama 8B Q6_K
model size params backend ngl test t/s
llama 8B Q6_K 6.14 GiB 8.03 B ROCm -1 pp128 961.02 ± 39.37
llama 8B Q6_K 6.14 GiB 8.03 B ROCm -1 pp256 1279.67 ± 15.06
llama 8B Q6_K 6.14 GiB 8.03 B ROCm -1 pp512 1393.37 ± 9.35
llama 8B Q6_K 6.14 GiB 8.03 B ROCm -1 pp1024 1378.50 ± 0.57
llama 8B Q6_K 6.14 GiB 8.03 B ROCm -1 pp2048 1341.89 ± 0.52
llama 8B Q6_K 6.14 GiB 8.03 B ROCm -1 pp128 @ d8192 746.54 ± 14.00
llama 8B Q6_K 6.14 GiB 8.03 B ROCm -1 pp256 @ d8192 924.78 ± 10.28
llama 8B Q6_K 6.14 GiB 8.03 B ROCm -1 pp512 @ d8192 983.12 ± 5.51
llama 8B Q6_K 6.14 GiB 8.03 B ROCm -1 pp1024 @ d8192 973.56 ± 0.17
llama 8B Q6_K 6.14 GiB 8.03 B ROCm -1 pp2048 @ d8192 954.36 ± 0.44
qwen35 27B Q6_K
model size params backend ngl test t/s
qwen35 27B Q6_K 21.49 GiB 27.32 B ROCm -1 pp128 324.60 ± 16.70
qwen35 27B Q6_K 21.49 GiB 27.32 B ROCm -1 pp256 380.59 ± 9.04
qwen35 27B Q6_K 21.49 GiB 27.32 B ROCm -1 pp512 414.32 ± 5.16
qwen35 27B Q6_K 21.49 GiB 27.32 B ROCm -1 pp1024 413.37 ± 0.40
qwen35 27B Q6_K 21.49 GiB 27.32 B ROCm -1 pp2048 409.68 ± 0.30
qwen35 27B Q6_K 21.49 GiB 27.32 B ROCm -1 pp128 @ d8192 302.18 ± 11.44
qwen35 27B Q6_K 21.49 GiB 27.32 B ROCm -1 pp256 @ d8192 348.23 ± 7.55
qwen35 27B Q6_K 21.49 GiB 27.32 B ROCm -1 pp512 @ d8192 376.80 ± 4.44
qwen35 27B Q6_K 21.49 GiB 27.32 B ROCm -1 pp1024 @ d8192 376.86 ± 0.05
qwen35 27B Q6_K 21.49 GiB 27.32 B ROCm -1 pp2048 @ d8192 374.57 ± 0.07

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES, AI was use to understand the codebase and write the automated sweep script. Final code was written by me.

@draetheus
draetheus requested a review from a team as a code owner August 22, 2026 17:41
@github-actions github-actions Bot added ggml changes relating to the ggml tensor library for machine learning CUDA Related to the CUDA backend labels Aug 22, 2026
@IMbackK IMbackK self-assigned this Aug 25, 2026
IMbackK
IMbackK reviewed Aug 25, 2026
View reviewed changes
Comment thread ggml/src/ggml-cuda/mmq.cuh
// J exceeds the per-expert width, wastes a full J-wide compute per tile. Use
// the per-channel width for selection only; the launch grid still uses
// ncols_max, which is a safe upper bound on any single expert's slots.
const int64_t ncols_sel = (GGML_CUDA_CC_IS_RDNA2(cc) && args.ids_dst != nullptr)

@IMbackK IMbackK Aug 25, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you test #24546 on rdna2 (enable it in mmq-config-rdna2.cuh) as an alterative to this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@IMbackK IMbackK IMbackK left review comments

At least 2 approving reviews are required to merge this pull request.

Assignees

@IMbackK IMbackK

Labels

CUDA Related to the CUDA backend ggml changes relating to the ggml tensor library for machine learning

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@draetheus @IMbackK