Q4_K_S: token2wav produces noise/non-speech output on Apple Silicon

[jasagiri]

Summary

On Apple Silicon (M3), the Q4_K_S quantized model (CosyVoice3-2512_Q4_K_S.gguf) produces output that sounds like noise/machine sound rather than intelligible speech. The LLM stage generates speech tokens successfully, but the final audio from token2wav (Flow + HiFT) is not recognizable as speech.

Environment

• Hardware: Apple Silicon M3 (MacBook Air)
• Model: Lourdle/Fun-CosyVoice3-0.5B-2512-GGUF (Q4_K_S, 551MB)
• Frontend: speech_tokenizer_v3.onnx + campplus.onnx
• Build: Metal + CPU (with PR feat: Apple Silicon support — SIMDe, Metal PAD beg-padding, op fallback, stop token fix #2 patches applied)

Symptoms

1. HiFT vocoder IM2COL memory explosion (long sequences)

For sequences with >~400 speech tokens, HiFT Conv1d layers produce IM2COL tensors with ne[1] > 0xFFFF (e.g. ne={896, 149761, 1, 1}). There are 27 such tensors per HiFT pass, each requiring ~200MB+, causing segfaults.

The current workaround (node->op = GGML_OP_NONE for oversized IM2COL) prevents the crash but completely corrupts vocoder output — producing a repeating 4-sample pattern:

[0.002363, -0.002809, 0.006419, -0.003742, 0.002363, -0.002809, ...]

2. LLM token generation instability

The Q4_K_S model generates wildly inconsistent speech token counts. For 13 single-sentence inputs with the same reference audio:

• 4 produced reasonable length (3-12s)
• 9 produced near-empty output (0.2-0.4s)

3. Non-speech output even for "working" lengths

Even when the LLM generates a reasonable number of tokens and IM2COL is not disabled (short sequences), the output waveform has speech-like statistical properties (zero-crossing rate, autocorrelation) but does not sound like speech to human listeners.

Analysis

Factor	Impact
Q4_K_S quantization (4-bit)	Model precision severely degraded. Stop token suppression partially helps but LLM remains unstable
HiFT IM2COL limit	Conv1d intermediate tensors explode for sequences >~400 tokens. Disabling IM2COL corrupts output
Metal/CPU mixed execution	Resolved via CPU-only workaround in token2wav (PR #2), but performance impact

Potential fixes

1. Higher precision quantization — Q8_0 or Q5_K_M should significantly improve LLM stability and vocoder quality. Is a Q8_0 GGUF available or can one be produced from the FP16 weights?

2. Chunked HiFT processing — Split speech_feat into overlapping chunks with proper Conv1d receptive field padding, run HiFT on each chunk, crossfade and concatenate. This would avoid the IM2COL memory explosion.

3. Streaming/tiled IM2COL — Implement IM2COL in tiles rather than materializing the full intermediate tensor. This is a ggml-level change.

Reproduction

# Build with PR #2 patches
cosyvoice-cli \
  --model CosyVoice3-2512_Q4_K_S.gguf \
  --speech-tokenizer speech_tokenizer_v3.onnx \
  --campplus campplus.onnx \
  --prompt-audio reference.wav \
  --prompt-text "reference transcript" \
  --text "Any Japanese text" \
  --output output.wav

# Analyze output
python3 -c "
import struct
with open('output.wav', 'rb') as f:
    data = f.read()
    idx = data.find(b'data') + 8
    floats = struct.unpack_from(f'<{(len(data)-idx)//4}f', data, idx)
    print(f'max={max(abs(f) for f in floats):.4f}')
    # Check for 4-sample repeat pattern (corrupted HiFT)
    if len(floats) > 200:
        is_loop = all(abs(floats[i]-floats[i+4])<0.0001 for i in range(100,120))
        print(f'corrupted={is_loop}')
"

Related

• PR feat: Apple Silicon support — SIMDe, Metal PAD beg-padding, op fallback, stop token fix #2: Apple Silicon support (SIMDe, PAD, op fallback, stop token fix, CPU-only token2wav)
• Issue Metal backend: unsupported op 'PAD' crash on Apple Silicon #3: Metal PAD/IM2COL crash report

[Lourdle]

I already implemented chunking for IM2COL from the start. See src/cosyvoice-graph.cpp.

I've tried a lot of different backends, but most of them are unstable when running HiFT. Only the Windows CUDA backend seems to be relatively stable. I suspect that this could be related to some issues in the upstream GGML project.

[jasagiri]

Update: Q8_0 model also fails — LLM sampler issue confirmed

Tested with CosyVoice3-2512_Q8_0.gguf (944MB). Results are essentially the same as Q4_K_S.

Root cause: LLM generates stop tokens too early

Debug output shows the LLM generates stop tokens within the first 5-10 iterations, regardless of min_token_text_ratio. The existing stop-token suppression mechanism is insufficient:

# Working case (text_len=5, min_len=10):
LLM: text_len=5, min_len=10, max_len=100
LLM: stop at n=61 → 2.4s ✓

# Failing case (text_len=21, min_len=42):
LLM: text_len=21, min_len=42, max_len=420
LLM: generated 5 tokens, suppressed 1 stops → 0.2s ✗

The suppress path zeros stop-token probabilities in the nucleus set and re-samples, but when the entire nucleus consists of stop tokens (sum == 0), it falls through and breaks. Adding retry loops (up to 3 retries) improved success rate from ~30% to ~50%, but the fundamental issue remains.

Statistics (Q8_0, 8 sentences, single-sentence input):

• 2/8 produced reasonable audio (2.9s, 3.3s)
• 6/8 produced near-empty output (0.16-0.44s)

Hypothesis

The quantized models may have degraded logit distributions that cause the sampler to strongly prefer stop tokens. The Python reference implementation likely handles this differently (perhaps with different sampling parameters, temperature, or repetition penalty).

Suggested investigation

1. Compare sampler parameters (temperature, top_k, top_p) between Python and C++ implementations
2. Check if the Python reference uses repetition penalty or frequency penalty to suppress stop tokens
3. Consider whether the nucleus_probs extraction is correctly implementing the Python sampling logic

[Lourdle]

Could you send the reference audio and its transcript? I wanna check if it works on CUDA.

[jasagiri]

Reference audio and new findings

Important discovery: the issue is reference-audio dependent

The original reference audio (a natural Japanese voice recording) causes the LLM to generate stop tokens very early, resulting in near-empty output ~70% of the time.

However, when using a macOS TTS-generated reference (Kyoko voice via say command), the LLM generates tokens stably — 8/8 sentences produced correct-length audio with Q8_0 model.

This suggests the issue is related to how the speech tokenizer + campplus frontend processes certain voice characteristics, leading to embeddings that put the LLM in a "stop-biased" state.

How to reproduce the reference audio

Since the original recording has copyright restrictions, here's how to generate a test reference:

# Generate reference with macOS TTS (Kyoko voice)
say -v Kyoko "みんな、人間のバグだって言ってくれんじゃん。これはテスト用の音声サンプルです。" -o reference.aiff
ffmpeg -i reference.aiff -ar 24000 -ac 1 -acodec pcm_s16le reference.wav

# This reference works stably with Q8_0:
cosyvoice-cli \
  --model CosyVoice3-2512_Q8_0.gguf \
  --speech-tokenizer speech_tokenizer_v3.onnx \
  --campplus campplus.onnx \
  --prompt-audio reference.wav \
  --prompt-text "みんな、人間のバグだって言ってくれんじゃん。これはテスト用の音声サンプルです。" \
  --text "最近の音声合成技術は本当にすごい進歩を遂げています。" \
  --output output.wav

For testing with a natural voice that triggers the early-stop issue, any short (5-10s) Japanese speech recording at 24kHz mono should work. The issue appears more frequently with:

• Natural/informal speech patterns
• Recordings with background noise or varying volume
• Non-studio quality recordings

Results with Kyoko reference (Q8_0)

Sentence	Tokens	Duration	Status
みなさん、こんにちは。…	188	7.6s	✓
最近の音声合成技術は…	129	5.2s	✓
人間の声をほぼ完璧に…	110	4.4s	✓
たった数秒の音声…	146	5.9s	✓
全く別のテキストを…	89	3.6s	✓
従来の音声合成では…	126	5.1s	✓
この技術を良い方向に…	110	4.4s	✓
今後の音声合成の進化に…	105	4.2s	✓

All 8 segments concatenated to 40.5s of audio. The generated speech is recognizable but still has a synthetic/mechanical quality — likely a limitation of the quantized model.

Remaining question

Could you test on CUDA with a natural voice recording to see if the early-stop issue is GGUF-quantization-specific or also occurs with the CUDA backend? If CUDA handles natural voice references stably, the issue is likely in the Metal/CPU execution path affecting logit computation.

[lui62233]

Hey @jasagiri, this is a really detailed breakdown. The IM2COL memory explosion for sequences >400 tokens is a classic ggml/Metal bottleneck when dealing with large 1D convolutions.

Regarding the '4-sample repeat pattern' you're seeing when disabling IM2COL: that's almost certainly because the convolution output is essentially zeroed or constant, and the subsequent HiFT layers are just amplifying a DC offset or a tiny rounding error into a periodic waveform.

I think your 'Chunked HiFT processing' idea is the most pragmatic path forward. If you can split the speech_feat into chunks (say, 256 tokens) with a small overlap (e.g., 32 tokens) to handle the receptive field of the Conv1d layers, you can bypass the 0xFFFF index limit entirely without needing a deep ggml-level rewrite of IM2COL.

As for the LLM instability with Q4_K_S, 4-bit is often too aggressive for speech tokens where precision in the latent space is critical for the vocoder. If you have the VRAM, trying a Q8_0 would be the first sanity check. If that works, we know the issue is purely quantization noise; if it doesn't, we're looking at a Metal/CPU synchronization issue in the LLM loop.

Would you be open to trying a chunking implementation for the HiFT pass? I can help sketch out the overlap/padding logic if that's useful.

[jasagiri]

@lui62233 Thanks for the analysis — you're right about the DC offset amplification when IM2COL is disabled. The 4-sample repeat pattern makes perfect sense as a degenerate Conv1d output.

Chunked HiFT — yes, happy to try this

Your suggested approach (256-token chunks with 32-token overlap) sounds reasonable. Before implementing, I want to nail down the receptive field sizes:

Questions for the implementation:

1. HiFT Conv1d kernel sizes — I see the graph has Conv1d layers with varying kernel sizes (from the IM2COL ne[0] values: 18, 192, 448, 704, 896). The largest kernel is 896, so the effective receptive field might be quite large once you stack multiple layers. Do you know the total receptive field of the HiFT network? That determines the minimum overlap.

2. Crossfade strategy — After processing overlapping chunks, we need to blend the overlap regions. Linear crossfade is simplest, but cosine or equal-power crossfade might give smoother transitions. Any preference?

3. Where to split — I assume we split speech_feat (the Flow output) rather than the raw speech tokens, since Flow's DiT steps are global. So the pipeline would be:

   LLM → all speech tokens
   Flow → full speech_feat (this is before IM2COL issues)
   HiFT → chunked processing of speech_feat
   

   Is that your understanding too?

Q8_0 update

Already tested Q8_0 — the LLM instability turned out to be reference-audio dependent, not quantization-related. With a clean TTS-generated reference (macOS Kyoko), Q8_0 produces stable output (8/8 success). With natural voice recordings, it fails ~70% of the time. This points to the speech tokenizer/campplus frontend producing embeddings that bias the LLM toward stop tokens for certain voice profiles.

I'll start sketching the chunked HiFT implementation. Any help with the overlap/padding logic is welcome.

[jasagiri]

Update: deeper investigation — two separate issues found

After extensive debugging, I've identified two distinct problems on Apple Silicon (CPU backend):

1. IM2COL chunking corrupts HiFT output for long sequences

When Conv1d::build_cgraph() triggers the chunking path (ne[1] > 0xFFFF), the output degrades to a repeating 4-sample pattern (near-silence, max_abs ≈ 0.006).

Verified by disabling chunking: with the if (im2col->ne[1] > 0xFFFF) check bypassed, the same long sequence (418 tokens, 838 speech_feat frames) produces output with proper signal levels (max_abs ≈ 0.072) and correct graph node count (1407 vs 1542 with chunking).

This confirms the chunking mechanism itself has an issue on the CPU backend — possibly related to how view_src/view_offs interact with ggml_backend_sched during graph splitting, or with the post-allocation GGML_OP_NONE conversion. On CUDA this likely works because the scheduler handles view tensors differently.

2. HiFT vocoder produces non-speech audio on CPU backend (regardless of chunking)

Even with chunking disabled and correct signal levels, the HiFT output sounds like mechanical noise rather than intelligible speech. This affects both short and long sequences, all quantization levels (Q4_K_S, Q8_0), and all reference audio types.

This suggests a deeper numerical issue in the CPU execution path — possibly in:

• ggml_stft/ggml_istft (magnitude/phase reconstruction)
• ggml_cumsum (phase accumulation in SineGen2, float32 precision)
• The source excitation path (harmonic + noise generation)

Current status

I've spent considerable time on this and the remaining issues appear to be in the ggml op implementations for the CPU backend, which would require significant effort to debug and fix. Unfortunately I don't have access to a CUDA machine to compare numerical outputs between backends.

The Apple Silicon build patches in PR #2 (SIMDe, Metal PAD, op fallback, stop token fix) are still valuable for getting the project to compile and run the LLM stage on Metal. I'll keep the PR updated with the review feedback, but I won't be able to tackle the HiFT CPU correctness issue in the near term.

If you have any suggestions for narrowing down the CPU backend issue (e.g., specific ggml ops to check, or intermediate tensor values to compare against CUDA output), I'm happy to run targeted tests.

Summary of findings for anyone investigating further

Component	Status	Notes
Build (SIMDe + ARM64)	✅ Working	PR #2
LLM (Metal)	✅ Working	Stop token fix included
token2wav Metal→CPU fallback	✅ Working	Avoids mixed execution segfault
IM2COL chunking (CPU)	❌ Broken	Produces 4-sample repeat; disabling chunking fixes signal level
HiFT vocoder (CPU)	❌ Non-speech output	Signal present but not intelligible speech; likely ggml op issue

[Lourdle]

Thanks for the incredibly detailed analysis and for your work on Apple Silicon support!

I want to share some findings from my side that perfectly align with your observations, and might give us a clue on where to look next.

1. The Quantization Impact (Q4 vs Q8) Your suspicion about quantization is spot on. In my tests (synthesizing Chinese text), Q8_0 sounds quite good, but the audio quality drops significantly from Q5 downwards. Q2_K is just pure noise. The HiFT vocoder is highly sensitive to precision loss. So the "machine sound / noise" you hear in Q4_K_S is heavily influenced by the 4-bit quantization itself.

2. Let's focus on Short Sentences & the HiFT Module first Before we tackle the im2col memory explosion (chunking) for long sequences, we should focus entirely on getting short sentences working perfectly. From my testing across different backends, the LLM and Flow modules usually run fine. The root cause of the corruption almost always lies within the HiFT module's operators.

3. A crucial clue: Uninitialized Memory / C++ Runtime issues To give you an idea of how fragile HiFT is right now: on Windows with CUDA, the Debug build works perfectly. The Release build produces corrupted output, unless I change the C/C++ Runtime Library to "Multi-threaded Debug DLL (/MDd)" or "Multi-threaded Debug (/MTd)" in Visual Studio. Furthermore, CUDA on Linux is still completely broken for me. This compiler/runtime-dependent behavior strongly suggests there is an uninitialized memory bug, an out-of-bounds array access, or a tensor alignment issue inside one of the GGML operators used by HiFT. The debug allocator likely zeroes out memory or adds padding that magically hides the bug. This same underlying C++ memory bug might be what is causing the Metal/CPU fallback to produce repeating 4-sample garbage patterns.

4. Language and Reference Audio (Suggestion) Regarding the LLM token generation instability: I noticed you are synthesizing Japanese text. CosyVoice's native zero-shot support for Japanese might not be as robust as it is for Chinese or English, especially under Q4 quantization. You also noticed that using a TTS-generated voice as the reference prompt is stable, but a natural human voice causes issues. This indicates the LLM might be struggling to extract acoustic features from complex natural prompts in a lower-precision state. 👉 Suggestion: For debugging purposes, could you try synthesizing a short English sentence with an English reference audio? English should be much more stable. This will help us isolate whether the issue is "model-level language instability" or a "backend computation bug".

Next Steps:

• Try testing with a short English prompt and text to see if the LLM token length stabilizes.
• Could you try generating and testing a Q8_0 GGUF for a very short sentence? This would help us completely separate "quantization noise" from "operator bugs" on Apple Silicon.
• Meanwhile, I will keep investigating the memory bugs on the CUDA/Linux side. If I can find and fix the uninitialized memory issue in the C++ code, it might automatically resolve the Metal/CPU crashes you are seeing.

Thanks again for digging into this! Let's conquer the short sentence HiFT operators first.

[jasagiri]

Test results: English short sentences with Q8_0

Tested per your suggestion — short English text, English TTS reference (macOS Samantha), Q8_0 model.

Setup

say -v Samantha "Hello, this is a test reference audio sample for voice cloning." -o ref.aiff
ffmpeg -i ref.aiff -ar 24000 -ac 1 -acodec pcm_s16le ref.wav

Results

Text	Tokens	Duration	Max abs	Audio quality
"The quick brown fox jumps over the lazy dog."	68	2.8s	0.707	❌ Not speech
"Hello world, this is a simple test."	76	3.1s	0.664	❌ Not speech
"Artificial intelligence is changing the world."	74	3.0s	0.990	❌ Not speech
"Today is a beautiful day for a walk in the park."	67	2.7s	0.990	❌ Not speech

LLM is working perfectly — stable token generation, no early stops, good token counts. The signal levels are strong (max abs up to 0.99).

But the output is not speech at all — it sounds like "shhhh-wooon shhhh-wooon", a repetitive mechanical/swooshing noise. Not degraded speech, not noisy speech — just a non-speech sound pattern.

Conclusion

This confirms the issue is not related to:

• Language (English has the same problem)
• Quantization level (Q8_0 with strong signal levels)
• Reference audio quality (clean TTS reference)
• LLM token generation (stable, correct length)
• IM2COL chunking (short sequences, no chunking triggered)

The problem is squarely in HiFT operator execution on the CPU backend. The "shhhh-wooon" pattern strongly suggests the STFT→magnitude/phase→ISTFT reconstruction is producing incorrect phase relationships, which would be consistent with your finding about uninitialized memory or alignment issues in the debug vs release build behavior.

I'll wait for your investigation on the CUDA/Linux side. If you find and fix the memory bug, I can immediately test on Apple Silicon to verify it resolves our issue too. In the meantime, PR #2 (build support + LLM fixes) is ready for review whenever convenient.

[jasagiri]

Root cause found: ggml_cont after ggml_stft zeroes out data

Added named intermediate tensor dumps to trace HiFT pipeline. Here's the smoking gun:

hift_s_stft:          ne={10081,9,2}, min=-0.1712, max=0.0425   ← STFT output: valid data
hift_s_stft (cont):   ne={10081,9,2}, min=0.0000,  max=0.0000   ← ggml_cont: ALL ZEROS
hift_s_stft (reshaped): ne={10081,18,1}, min=0.0000, max=0.0000  ← stays zero

The ggml_cont(ctx, s_stft) at cosyvoice-graph.cpp:711 is producing an all-zero tensor. Everything downstream (source excitation mixing, ISTFT reconstruction) operates on zeros, producing the mechanical noise pattern.

Why ggml_cont fails

ggml_stft returns a permuted view of the FFT output:

// fft.cpp:1129-1130
result = ggml_view_3d(ctx, result, fctx->nfft / 2 + 1, n_frames, 2, ...);
return ggml_permute(ctx, result, 1, 0, 2, 3);

The resulting tensor is non-contiguous (permuted). ggml_cont should copy it to contiguous memory, but the CPU implementation appears to mishandle the stride pattern from the FFT output's internal layout.

This is likely the same bug that manifests differently on CUDA (where ggml_cont may use a different code path). It would also explain the Debug vs Release difference on Windows — the memory layout/alignment of intermediate FFT buffers may differ.

Suggested fix

Option A: Fix ggml_cont to handle the specific stride pattern from ggml_stft output.

Option B: Avoid the cont+reshape by doing the reshape directly on the permuted tensor, or restructure the STFT output to be contiguous before returning.

Option C: Replace ggml_cont with an explicit element-wise copy loop that handles the permuted layout.

I can test any proposed fix immediately on Apple Silicon. This is likely the single fix needed to get speech output working.

[jasagiri]

Correction: root cause is NOT ggml_cont — it's Metal→CPU weight access

My previous comment about ggml_cont zeroing STFT data was incorrect. The zero values were caused by a more fundamental issue:

The actual root cause

All Flow+HiFT model weights are allocated on the Metal buffer (via ggml_backend_get_default_buffer_type(backend.get()) in cosyvoice-loader.cpp:388). When token2wav routes all computation to CPU backend using set_graph_backend(..., cpu_backend.get(), cpu_backend.get()), the CPU compute kernels read the weight tensors' data pointers — but these point into Metal GPU memory, which is inaccessible from CPU code. The result is all-zero weights, producing mechanical noise.

Evidence

• conv_post.weight (type=f16): reading via reinterpret_cast<float*>(node->data) shows all zeros
• Same weight read via ggml_backend_tensor_get() (which handles cross-backend reads) shows correct values (max_abs=0.003679)
• conv_pre.bias (type=f32, allocated on CPU): reads correctly with max_abs=0.392

Why Metal mixed execution segfaults

When using the original set_graph_backend(gf, sched.get(), backend.get(), cpu_backend.get()), unsupported ops (PAD, IM2COL, CUSTOM) are routed to CPU while supported ops stay on Metal. The ggml_backend_sched_graph_compute segfaults during tensor data transfer between backends.

Proposed fix

The cleanest fix is to allocate Flow+HiFT weights on CPU buffer while keeping LLM weights on Metal. In cosyvoice-loader.cpp, the weight allocation loop at line 430-435 should use CPU buffer type for Flow/HiFT tensors:

// Determine buffer type based on tensor ownership
auto tensor_buft = is_llm_tensor(name) ? buft : ggml_backend_cpu_buffer_type();

This way LLM runs on Metal with Metal-allocated weights, and token2wav runs entirely on CPU with CPU-allocated weights. No cross-backend copies needed.

An alternative is to fix the Metal→CPU segfault in ggml_backend_sched, which would allow mixed execution. But that's a deeper ggml-level fix.

I can implement the split-buffer approach if you agree it's the right direction. It would add ~20 lines to the loader.

[Lourdle]

Huge breakthrough on my end! I think I have found the root cause of the noise across all platforms.

I finally figured out why my Windows build was only working in Debug mode. It turns out it only generates clean audio when linked specifically against ucrtbased.dll (the Windows Debug Universal C Runtime). If I link to the standard release runtime, it outputs the exact same noise.

The Hypothesis: A Buffer Overflow (Out-of-bounds write) I strongly suspect there is a buffer overflow happening somewhere during the graph evaluation (most likely inside the HiFT module). We are allocating a tensor/buffer that is slightly too small, and writing past its end.

• On Windows Debug, the memory allocator automatically adds extra padding (guard bytes) around every allocation. This padding accidentally "absorbs" the overflow, preventing memory corruption, so the audio sounds fine.
• On Linux (which uses a different debug allocator) and on your macOS Release build, there is no such padding. The out-of-bounds write corrupts adjacent tensor data in memory, which leads to the severe noise.