q4k mmq optimizations by liangliangchang · Pull Request #80 · AMD-Ecosystem/llama.cpp

liangliangchang · 2026-07-31T22:58:56Z

Overview

MUL_MAT shape (m × n × k)	Baseline	`lichang.q4k-opt`	Improvement	`q4k-opt` throughput
1024 × 128 × 4096	85.54 µs	65.42 µs	+23.52%	16.41 TFLOPS
5376 × 128 × 2048	152.08 µs	142.75 µs	+6.13%	19.74 TFLOPS
4096 × 128 × 4096	216.61 µs	201.98 µs	+6.75%	21.26 TFLOPS
8192 × 128 × 4096	360.98 µs	346.59 µs	+3.99%	24.78 TFLOPS
12288 × 128 × 4096	527.31 µs	504.85 µs	+4.26%	25.52 TFLOPS
4096 × 512 × 14336	2549.82 µs	2490.45 µs	+2.33%	24.14 TFLOPS

Test	Group	TTFT improvement	Decode improvement	Baseline	Candidate
Qwen2.5-0.5B-Instruct_Q4_K_M_GGUF_128	standard_128	1.88%	-0.03%	PASS	PASS
Qwen2.5-0.5B-Instruct_Q4_K_M_GGUF_3968	standard_long_context	0.26%	-0.26%	PASS	PASS
Qwen2.5-3B-Instruct_Q4_K_M_GGUF_128	standard_128	5.38%	0.01%	PASS	PASS
Qwen2.5-3B-Instruct_Q4_K_M_GGUF_3968	standard_long_context	2.06%	-0.13%	PASS	PASS
Qwen2.5-7B-Instruct_Q4_K_M_GGUF_128	standard_128	1.61%	0.46%	PASS	PASS
Qwen2.5-7B-Instruct_Q4_K_M_GGUF_3968	standard_long_context	0.79%	-0.15%	PASS	PASS
Qwen3-1.7B_Q4_K_M_GGUF_128	standard_128	1.67%	-0.03%	PASS	PASS
Qwen3-1.7B_Q4_K_M_GGUF_3968	standard_long_context	1.09%	-0.10%	PASS	PASS
Qwen3-4B_Q4_K_M_GGUF_128	standard_128	2.39%	not rerun	PASS	PASS
Qwen3-4B_Q4_K_M_GGUF_3968	standard_long_context	1.56%	0.27%	PASS	PASS
Qwen3-8B_Q4_K_M_GGUF_128	standard_128	1.87%	0.09%	PASS	PASS
Qwen3-8B_Q4_K_M_GGUF_3968	standard_long_context	0.14%	0.18%	PASS	PASS
Qwen3.5-4B_Q4_K_M_GGUF_128	standard_128	1.57%	0.03%	PASS	PASS
Qwen3.5-9B_Q4_K_M_GGUF_128	standard_128	3.42%	0.06%	PASS	PASS
Llama-2-7B_Q4_K_M_GGUF_128	standard_128	1.56%	3.36%	PASS	PASS
Llama-2-7B_Q4_K_M_GGUF_1920	standard_long_context	0.37%	0.03%	PASS	PASS
Gemma-2B_Q4_K_M_GGUF_128	standard_128	2.32%	-0.08%	PASS	PASS
Gemma-2B_Q4_K_M_GGUF_8000	standard_long_context	-0.59%	0.01%	PASS	PASS
SmolLM2-1.7B-Instruct_Q4_K_M_GGUF_128	standard_128	1.18%	-0.05%	PASS	PASS
SmolLM2-1.7B-Instruct_Q4_K_M_GGUF_8000	standard_long_context	0.32%	-0.07%	PASS	PASS
Llama-3.1-8B-Instruct_Q4_K_M_GGUF_128	standard_128	-0.77%	0.19%	PASS	PASS
Qwen3-30B-A3B-Instruct-2507_Q4_K_M_GGUF_128	standard_128	0.49%	not rerun	PASS	PASS
Gemma-4-E2B-IT_Q4_K_M_GGUF_128	standard_128	2.76%	-0.07%	PASS	PASS
Gemma-4-E2B-IT_Q4_K_M_GGUF_3968	standard_long_context	0.31%	0.06%	PASS	PASS
Qwen3.5-35B-A3B_Q4_K_M_GGUF_128	standard_128	3.67%	not rerun	PASS	PASS
Qwen3.6-35B-A3B_Q4_K_M_GGUF_128	standard_128	0.24%	not rerun	PASS	PASS
Qwen3.6-35B-A3B_Q4_K_M_GGUF_4096	standard_long_context	0.16%	-0.02%	PASS	PASS
Gemma-3-4B-IT_VLM	standard_other	0.38%	-0.07%	PASS	PASS
Qwen2.5-VL-3B-Instruct_VLM	standard_other	0.30%	-0.00%	PASS	PASS
Qwen2.5-VL-7B-Instruct_VLM	standard_other	0.04%	-0.01%	PASS	PASS
Qwen3-VL-4B-Instruct_VLM	standard_other	0.41%	0.09%	PASS	PASS
Gemma-4-26B-A4B-IT_VLM	standard_other	0.14%	-0.01%	PASS	PASS
Gemma-4-31B-IT_VLM	standard_other	0.68%	0.06%	PASS	PASS
MiniCPM-V-2_6_VLM	standard_other	0.65%	0.06%	PASS	PASS
MiniCPM-o-2_6_VLM	standard_other	0.75%	0.15%	PASS	PASS
Janus-Pro-1B_VLM	standard_other	2.42%	-0.00%	PASS	PASS
Janus-Pro-7B_VLM	standard_other	1.43%	-0.05%	PASS	PASS
Qwen3.6-27B_Q4_K_M_GGUF_4096	standard_long_context	0.61%	-0.03%	PASS	PASS
GLM-4.7-Flash_Q4_K_M_GGUF_128	standard_128	-0.15%	0.11%	PASS	PASS
Gemma-4-12B-it_Q4_K_M_GGUF_128	standard_128	-0.04%	-0.03%	PASS	PASS
Gemma-3-12B-IT_Q4_K_M_GGUF_VLM	standard_other	0.81%	0.10%	PASS	PASS
Gemma-3-12B-IT_Q4_K_M_GGUF_4096	standard_long_context	0.85%	0.10%	PASS	PASS
Qwen3.5-4B_Q4_0_GGUF_baseline_prose	mtp_baseline	-0.93%	-0.27%	PASS	PASS
Qwen3.5-4B_Q4_0_GGUF_mtp_prose	mtp_verify	-0.15%	0.18%	PASS	PASS
Qwen3.5-4B_Q4_0_GGUF_baseline_code	mtp_baseline	-1.49%	-0.29%	PASS	PASS
Qwen3.5-4B_Q4_0_GGUF_mtp_code	mtp_verify	-1.34%	0.30%	PASS	PASS
Qwen3.5-4B_Q4_0_GGUF_baseline_list	mtp_baseline	-1.37%	-0.02%	PASS	PASS
Qwen3.5-4B_Q4_0_GGUF_mtp_list	mtp_verify	-0.89%	0.25%	PASS	PASS
Qwen3.5-4B_Q4_K_M_GGUF_baseline_prose	mtp_baseline	-0.95%	-0.06%	PASS	PASS
Qwen3.5-4B_Q4_K_M_GGUF_mtp_prose	mtp_verify	-1.52%	-0.07%	PASS	PASS
Qwen3.5-4B_Q4_K_M_GGUF_baseline_code	mtp_baseline	-1.81%	-0.06%	PASS	PASS
Qwen3.5-4B_Q4_K_M_GGUF_mtp_code	mtp_verify	-1.70%	-0.16%	PASS	PASS
Qwen3.5-4B_Q4_K_M_GGUF_baseline_list	mtp_baseline	-1.73%	-0.17%	PASS	PASS
Qwen3.5-4B_Q4_K_M_GGUF_mtp_list	mtp_verify	0.45%	-0.43%	PASS	PASS
Qwen3.5-9B_Q4_0_GGUF_baseline_prose	mtp_baseline	-0.25%	0.01%	PASS	PASS
Qwen3.5-9B_Q4_0_GGUF_mtp_prose	mtp_verify	0.01%	0.01%	PASS	PASS
Qwen3.5-9B_Q4_0_GGUF_baseline_code	mtp_baseline	0.86%	0.01%	PASS	PASS
Qwen3.5-9B_Q4_0_GGUF_mtp_code	mtp_verify	0.17%	-0.03%	PASS	PASS
Qwen3.5-9B_Q4_0_GGUF_baseline_list	mtp_baseline	-0.32%	-0.01%	PASS	PASS
Qwen3.5-9B_Q4_0_GGUF_mtp_list	mtp_verify	-0.68%	0.30%	PASS	PASS
Qwen3.5-9B_Q4_K_M_GGUF_baseline_prose	mtp_baseline	0.93%	0.04%	PASS	PASS
Qwen3.5-9B_Q4_K_M_GGUF_mtp_prose	mtp_verify	1.09%	0.09%	PASS	PASS
Qwen3.5-9B_Q4_K_M_GGUF_baseline_code	mtp_baseline	1.19%	0.04%	PASS	PASS
Qwen3.5-9B_Q4_K_M_GGUF_mtp_code	mtp_verify	0.38%	-0.07%	PASS	PASS
Qwen3.5-9B_Q4_K_M_GGUF_baseline_list	mtp_baseline	0.53%	0.05%	PASS	PASS
Qwen3.5-9B_Q4_K_M_GGUF_mtp_list	mtp_verify	-0.65%	-0.42%	PASS	PASS
Qwen3.6-27B_Q4_0_GGUF_baseline_prose	mtp_baseline	0.29%	0.01%	PASS	PASS
Qwen3.6-27B_Q4_0_GGUF_mtp_prose	mtp_verify	-0.17%	0.04%	PASS	PASS
Qwen3.6-27B_Q4_0_GGUF_baseline_code	mtp_baseline	-0.49%	-0.01%	PASS	PASS
Qwen3.6-27B_Q4_0_GGUF_mtp_code	mtp_verify	0.94%	-0.02%	PASS	PASS
Qwen3.6-27B_Q4_0_GGUF_baseline_list	mtp_baseline	0.21%	0.06%	PASS	PASS
Qwen3.6-27B_Q4_0_GGUF_mtp_list	mtp_verify	-0.00%	-0.11%	PASS	PASS
Qwen3.6-27B_Q4_K_M_GGUF_baseline_prose	mtp_baseline	0.79%	0.22%	PASS	PASS
Qwen3.6-27B_Q4_K_M_GGUF_mtp_prose	mtp_verify	0.96%	-0.37%	PASS	PASS
Qwen3.6-27B_Q4_K_M_GGUF_baseline_code	mtp_baseline	0.09%	-0.09%	PASS	PASS
Qwen3.6-27B_Q4_K_M_GGUF_mtp_code	mtp_verify	0.13%	-0.43%	PASS	PASS
Qwen3.6-27B_Q4_K_M_GGUF_baseline_list	mtp_baseline	1.31%	-0.17%	PASS	PASS
Qwen3.6-27B_Q4_K_M_GGUF_mtp_list	mtp_verify	-0.97%	-0.13%	PASS	PASS
Qwen3.6-35B-A3B_Q4_K_M_GGUF_baseline_prose	mtp_baseline	3.48%	0.03%	PASS	PASS
Qwen3.6-35B-A3B_Q4_K_M_GGUF_mtp_prose	mtp_verify	0.35%	-0.12%	PASS	PASS
Qwen3.6-35B-A3B_Q4_K_M_GGUF_baseline_code	mtp_baseline	1.37%	0.06%	PASS	PASS
Qwen3.6-35B-A3B_Q4_K_M_GGUF_mtp_code	mtp_verify	-0.30%	-0.06%	PASS	PASS
Qwen3.6-35B-A3B_Q4_K_M_GGUF_baseline_list	mtp_baseline	1.30%	-0.01%	PASS	PASS
Qwen3.6-35B-A3B_Q4_K_M_GGUF_mtp_list	mtp_verify	0.31%	-0.66%	PASS	PASS

Additional information

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure:

Pipeline Q4_K tile loads with split WMMA dequantization and select the measured tile width by matrix shape. Assisted-by: GPT-5.6 Sol Co-authored-by: Cursor <cursoragent@cursor.com>

Match the refactored 32-row warp layout and limit the specialized J64 path to the small shape where it provides a stable gain. Assisted-by: GPT-5.6 Sol Co-authored-by: Cursor <cursoragent@cursor.com>

Use the generic vec-dot with J64 two-row waves and J128 one-row waves, retaining the J64 prefetch loop for the best measured shape coverage. Assisted-by: GPT-5.6 Sol Co-authored-by: Cursor <cursoragent@cursor.com>

Avoid overriding MoE's per-expert tile selection with the dense Q4_K pipeline policy. Co-authored-by: Cursor <cursoragent@cursor.com>

Keep production dispatch shape-driven and avoid exposing an experimental tile-width environment knob. Co-authored-by: Cursor <cursoragent@cursor.com>

Reuse the validated next-weight pipeline for J128 to improve large Q4_K MMQ throughput without retaining experimental pacing controls. Co-authored-by: Cursor <cursoragent@cursor.com>

Separate low and high nibble WMMA phases across J128 tiles to retain the faster dequantization schedule with next-weight prefetch. Co-authored-by: Cursor <cursoragent@cursor.com>

liangliangchang and others added 5 commits July 31, 2026 13:18

HIP: optimize RDNA3.5 Q4_K MMQ pipeline

2815b3e

Pipeline Q4_K tile loads with split WMMA dequantization and select the measured tile width by matrix shape. Assisted-by: GPT-5.6 Sol Co-authored-by: Cursor <cursoragent@cursor.com>

HIP: fix RDNA3.5 Q4_K pipeline integration

a06b975

Match the refactored 32-row warp layout and limit the specialized J64 path to the small shape where it provides a stable gain. Assisted-by: GPT-5.6 Sol Co-authored-by: Cursor <cursoragent@cursor.com>

HIP: tune Q4_K wave mapping by tile width

c83bdac

Use the generic vec-dot with J64 two-row waves and J128 one-row waves, retaining the J64 prefetch loop for the best measured shape coverage. Assisted-by: GPT-5.6 Sol Co-authored-by: Cursor <cursoragent@cursor.com>

HIP: preserve expert-aware Q4_K MMQ tiling

bec4a25

Avoid overriding MoE's per-expert tile selection with the dense Q4_K pipeline policy. Co-authored-by: Cursor <cursoragent@cursor.com>

HIP: remove Q4_K MMQ test override

f1cd056

Keep production dispatch shape-driven and avoid exposing an experimental tile-width environment knob. Co-authored-by: Cursor <cursoragent@cursor.com>

liangliangchang marked this pull request as ready for review August 12, 2026 17:56

liangliangchang and others added 2 commits August 12, 2026 13:11

HIP: extend RDNA3.5 Q4_K prefetch to J128

f602551

Reuse the validated next-weight pipeline for J128 to improve large Q4_K MMQ throughput without retaining experimental pacing controls. Co-authored-by: Cursor <cursoragent@cursor.com>

HIP: batch RDNA3.5 Q4_K J128 WMMA phases

96fc67b

Separate low and high nibble WMMA phases across J128 tiles to retain the faster dequantization schedule with next-weight prefetch. Co-authored-by: Cursor <cursoragent@cursor.com>

liangliangchang requested review from Annieren, jimw567, mgehre-amd and roberteg16 August 13, 2026 23:57

Navigation Menu

Uh oh!

q4k mmq optimizations - #80

q4k mmq optimizations#80
liangliangchang wants to merge 7 commits into
gfx11from
lichang.q4k-opt

liangliangchang commented Jul 31, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

liangliangchang commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Additional information

Requirements

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

liangliangchang commented Jul 31, 2026 •

edited

Loading