# Accelerating MoE Inference: llama.cpp Integrates CUDA Kernel Pipelining for Multi-Token Prediction > Enrolling the quantized MoE matrix-vector multiplication kernel into the execution pipeline yields a consistent 5-6% throughput increase for speculative decoding. **Published:** June 05, 2026 **Author:** PSEEDR Editorial **Category:** stack **Content tier:** free **Accessible for free:** true **Editorial format:** analysis **News quality eligible:** true **Source count:** 1 **Word count:** 1068 **Tags:** llama.cpp, CUDA, Mixture of Experts (MoE), Multi-Token Prediction (MTP), Speculative Decoding, Model Optimization **Canonical URL:** https://pseedr.com/stack/accelerating-moe-inference-llamacpp-integrates-cuda-kernel-pipelining-for-multi- --- The latest b9521 release from github-llamacpp-releases introduces a targeted CUDA optimization that accelerates Multi-Token Prediction (MTP) for Mixture of Experts (MoE) models. By pipelining the quantized matrix-vector multiplication kernels, llama.cpp reduces kernel launch overhead and memory bottlenecks, signaling a critical maturation in how local inference engines handle complex speculative decoding architectures. The latest [b9521 release from github-llamacpp-releases](https://github.com/ggml-org/llama.cpp/releases/tag/b9521) introduces a targeted CUDA optimization that accelerates Multi-Token Prediction (MTP) for Mixture of Experts (MoE) models. By pipelining the quantized matrix-vector multiplication kernels, the project reduces kernel launch overhead and memory bottlenecks, signaling a critical maturation in how local inference engines handle complex speculative decoding architectures. ## The Mechanics of MoE Kernel Pipelining At the core of this update is the enrollment of the `mul_mat_vec_q_moe` kernel into the PDL (Pipelined Draft Loading) execution path within the llama.cpp CUDA backend. Mixture of Experts architectures inherently complicate standard inference pipelines because tokens must be dynamically routed to specific feed-forward network experts. When combined with quantization-which requires dequantizing weights on the fly-and Multi-Token Prediction, the overhead of launching individual CUDA kernels for matrix-vector multiplication becomes a significant bottleneck. Matrix-vector multiplication (`mul_mat_vec`) operations dominate the execution profile of Large Language Models during the decoding phase, where the effective batch size is small. By shifting this specific MoE kernel into a pipelined execution model, the engine can overlap computation with memory access more effectively. In speculative decoding setups, where a draft model or an MTP head predicts multiple future tokens simultaneously, the system must rapidly evaluate these drafts against the target model. Pipelining the MoE operations ensures that the GPU's compute units remain saturated, minimizing the idle time that typically occurs when waiting for the next set of expert weights to be loaded from VRAM. ## Benchmark Validation and Throughput Gains The performance impact of this micro-optimization is highly consistent across various generation tasks. Validation was performed using a Qwen 3.6 35B MoE model (specifically the `Qwen3.6-35B-A3B-UD-Q4_K_M.gguf` variant) configured for MTP speculative drafting. The server was launched with parameters enforcing a draft maximum of two tokens (`--spec-draft-n-max 2`) and utilizing Flash Attention (`-fa on`). Testing across a suite of standard prompts revealed a reliable 4.5% to 6.0% increase in generation throughput, measured in tokens per second (tok/s). Notable improvements include: * **Summarization:** Throughput increased from 226.6 tok/s to 240.2 tok/s, representing an approximate 6.0% speedup. * **Factual QA:** Performance improved from 225.1 tok/s to 238.5 tok/s, a 5.9% gain. * **C++ Code Generation:** Generation speed rose from 212.8 tok/s to 224.6 tok/s, yielding a 5.5% enhancement. * **Stepwise Mathematics:** Throughput climbed from 209.2 tok/s to 221.7 tok/s, marking a 6.0% increase. Crucially, the benchmark logs confirm that the draft acceptance rates (`rate`) and accuracy metrics (`acc`) remained identical before and after the update. This indicates that the throughput gains are strictly the result of computational efficiency rather than algorithmic degradation or changes to the speculative acceptance thresholds. ## Implications for Local Speculative Inference As speculative decoding and Multi-Token Prediction transition from experimental features to standard requirements for accelerating LLM inference, optimizing the underlying CUDA kernels is paramount. MoE models are notoriously memory bandwidth bound; their sparse activation means that while compute requirements per token are lower than dense models of equivalent parameter count, the memory subsystem is heavily taxed by the need to constantly fetch different expert weights. This update directly addresses that friction. By optimizing the quantized MoE kernels for pipelined execution, llama.cpp makes running complex, 35-billion-parameter models highly viable on local workstation hardware. Achieving sustained generation speeds in excess of 220 tokens per second for a model of this scale demonstrates the efficacy of combining MTP with aggressive low-level kernel optimization. For enterprise deployments and local developers, this translates to faster time-to-first-token and higher overall throughput without requiring a corresponding upgrade in GPU hardware. Furthermore, this optimization highlights a broader trend in the open-weight ecosystem. As model architectures stabilize around MoE and MTP paradigms, the engineering focus is shifting toward squeezing maximum performance out of the hardware through kernel fusion, pipelining, and memory management. The ability to overlap operations-as noted in the release's reference to overlapping with subsequent kernels-is critical for maintaining high GPU utilization and delaying hardware obsolescence for local AI practitioners. ## Limitations and Hardware Ambiguities While the performance gains are clearly documented, the release notes leave several contextual gaps that complicate broader extrapolation. The exact definition of "PDL" within the llama.cpp CUDA backend is not explicitly detailed in the source, though it likely refers to Pipelined Draft Loading or a Pipelined Dequantization Loop. Without explicit documentation, developers looking to port or adapt this pipeline logic to other backends, such as AMD's ROCm or Apple's Metal, face a steeper learning curve. Additionally, the hardware specifications of the "B4500" system used for benchmarking remain ambiguous. This could refer to a specific enterprise GPU, a specialized internal testbed, or potentially a typographical error for a known hardware configuration. The release also mentions boosting MTP performance on "BW," which likely stands for Bandwidth, but could also refer to a specific hardware platform or internal metric. Because MoE performance is highly dependent on memory bandwidth, understanding the exact VRAM speed and bus width of the test system is crucial for predicting how these gains will scale on consumer-grade hardware, such as NVIDIA RTX 4090s or Mac Studio systems. ## Synthesis The integration of the `mul_mat_vec_q_moe` kernel into the execution pipeline represents a highly effective micro-optimization for local LLM inference. By addressing the specific bottlenecks associated with quantized Mixture of Experts models and Multi-Token Prediction, llama.cpp has secured a measurable 5-6% throughput increase without sacrificing generation quality. As the open-source community continues to push the boundaries of what is possible on local hardware, these targeted CUDA optimizations will remain the primary driver for achieving data-center-level inference speeds on workstation machines. ### Key Takeaways * llama.cpp has optimized its CUDA backend by enrolling the quantized MoE matrix-vector multiplication kernel into its pipelined execution path. * The optimization yields a consistent 4.5% to 6% increase in tokens per second across various generation tasks without altering draft acceptance rates. * Benchmarking on a Qwen 3.6 35B MoE model demonstrated throughput exceeding 240 tokens per second for summarization tasks. * The update highlights the critical role of kernel pipelining in mitigating the memory bandwidth bottlenecks inherent to Mixture of Experts architectures. --- ## Sources - https://github.com/ggml-org/llama.cpp/releases/tag/b9521