sycl : fix Q8_0/k-quant reorder get_rows (garbage output with GGML_SYCL_ENABLE_OPT) - #3964
Open
xuhancn wants to merge 6 commits into
Open
sycl : fix Q8_0/k-quant reorder get_rows (garbage output with GGML_SYCL_ENABLE_OPT)#3964xuhancn wants to merge 6 commits into
xuhancn wants to merge 6 commits into
Conversation
* add /Zi + /DEBUG so cdb resolves frames in ggml-sycl.dll
* use quantize_f instead of hardcoded quantize_q8_1 in mul_mat * restore Q8_0 in the reorder support lists * fallback mul_mat_q to dequantize when weight is reordered * add debug prints to isolate the reorder issue
* k_get_rows_reorder template for Q8_0/Q4_0 (per-element) * block kernels for Q3_K/Q4_K/Q5_K/Q6_K (32/64 threads) * dispatch reorder path when src0 is reordered
xuhancn
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found while working on whisper_xpu (https://github.com/xuhancn/whisper_xpu) (a SYCL fork of whisper.cpp for Intel Arc GPU). Q8_0 models gave garbage output on GPU when the reorder optimization was on (GGML_SYCL_ENABLE_OPT=1, the default), but worked when it was off.
Root cause: opt_for_reorder rewrites quantized weights in place into a split layout (qs | scales | d) for the MMVQ mul_mat path. But ggml_sycl_op_get_rows (token embedding lookup) still dequantized with the standard interleaved block layout, reading misaligned data
from the split buffer and producing NaN. The NaN propagated GET_ROWS -> ADD -> NORM -> MUL -> ADD into the mul_mat_vec_q activation, leaving some logits zero and letting argmax pick random tokens.
Fix: reorder-aware path in ggml_sycl_op_get_rows. When src0->extra->optimized_feature.reorder is set, index the split layout directly:
Also fixed the mul_mat quantize path to use quantize_f instead of the hardcoded quantize_q8_1 (so the SoA quantizer is dispatched), and made mul_mat_q fall back to dequantize when the weight was already reordered.
Tests: standalone SYCL UTs for the reorder get_rows of Q3_K/Q4_K/Q5_K/Q6_K (host golden from dequantize_row_q*_K) — all pass; one caught a Q3_K thread-count mistake (128 vs 64). Lives at https://github.com/xuhancn/whisper_xpu/tree/main/tests (superproject PR #43
merged with the UTs: xuhancn/whisper_xpu#43). Q8_0 verified end-to-end with whisper-cli on large-v3-turbo-q8_0 and q4_0 tiny models: correct text.
Disclosure: AI tools were used to help write the kernel boilerplate and run the tests; the root-cause isolation, fix design, and verification were done by hand.