perf(cuda): full-offload MoE decode ~3.4× behind llama.cpp — OLMoE 126 vs 426 t/s (on-GPU experts, kernel-bound)

[pekkah]

Part of #405 (README vs llama.cpp CUDA benchmark sweep).

OLMoE-1B-7B is the only model in the sweep that lags badly on both axes. Prefill (176×) is #406; this issue is the decode gap.

Measured (RTX 4070 Ti, llama.cpp CUDA b9529, full offload)

	README (sharpi)	llama.cpp -ngl 99 tg128	gap
decode	126.0 t/s	426.1 t/s	3.4×

For contrast, the dense full-offload models decode within ~1.1–1.25× of llama.cpp (Qwen3-8B 74.5 vs 91, SmolLM2 268 vs 334, Gemma4-12B 54.1 vs 57). So 3.4× is specific to the MoE expert path, not a general decode deficiency — and unlike the 35B GDN hybrids, OLMoE's experts are resident on the GPU (4 GB model, -g -1), so this is not RAM-streaming bound; it's kernel efficiency.

Hypothesis (needs profiling)

OLMoE decode does a per-token routed-expert FFN (64 experts / 8 active, per-channel QK-norm, norm_topk_prob=false). The likely cost is the on-GPU expert matvec issued as 8 separate small GEMV/matvec launches per layer vs llama.cpp's fused mul_mat_id / mul_mat_vec_q over the expert tensors. Candidate levers (confirm with nsys/ncu first — see the measurement discipline in #400/#388):

1. Fused expert matvec — one mul_mat_id-style kernel that gathers the 8 selected expert rows and does the matvec in a single launch (vs per-expert launches), raising occupancy and cutting launch overhead.
2. Q8_1 / dp4a expert matvec — ensure the on-GPU expert decode matvec uses the int8 dp4a path (same one the dense trunk uses, llm_matvec_*_dp4a) rather than an F32/fp16 fallback.
3. Router on GPU — fold the router matvec + top-k into the decode trunk if it's currently host-side.

Acceptance

Profile decode first (the dense models are already near-ceiling, so the budget is entirely in the expert phase), then target the dominant kernel. Argmax-stable is acceptable. Re-measure vs tg128 426.

References

• OLMoE decode path: search _isMoE / expert FFN in CudaForwardPass + CudaBackend MoE matvec kernels
• Profiling discipline + nsys/ncu gotchas: perf(cuda): Carnice decode toward llama.cpp ~58 t/s — GPU trunk decode kernels are the lever (6.17ms vs ~1ms/tok) #400, Carnice prefill toward llama.cpp 676 t/s — per-layer rewrite levers (op-offload follow-up to #387) #388, and the project_decode_carnice_cpumoe notes (nsys avg lies; use ncu isolated)