The recent llama.cpp b9654 release introduces a critical post-decode callback mechanism to its Multi-Token Multi-Domain (MTMD) framework alongside an expanded matrix of hardware build targets. This update highlights the project's rapid evolution in speculative decoding capabilities and its aggressive pursuit of day-one compatibility with cutting-edge runtimes like CUDA 13 and specialized domestic architectures such as Huawei's Ascend.
\nThe most structurally significant update in this release is the integration of a post-decode callback within the MTMD implementation (pull request #24645). Multi-Token Multi-Domain (MTMD) is foundational to llama.cpp's approach to speculative decoding and drafting, where a smaller, highly efficient draft model predicts multiple future tokens, which are subsequently verified by a larger target model in a single forward pass. Historically, intervening in this highly optimized loop required invasive modifications to the core C++ inference engine.
By exposing a post-decode callback, the maintainers have decoupled the generation logic from the execution pipeline. Programmatically, this allows developers to inject custom logic immediately after a token (or sequence of tokens) is decoded but before the next iteration of the attention mechanism begins. The applications for this are extensive. Developers deploying local LLMs on resource-constrained edge devices can now implement dynamic early-stopping criteria, enforce strict grammar or schema constraints (such as guaranteed JSON outputs), or apply real-time safety filters without incurring the latency overhead of post-processing an entire generated sequence. Furthermore, this callback mechanism provides the necessary hooks for advanced watermarking techniques, where token probabilities are subtly altered during generation to embed cryptographic signatures.
Beyond architectural refinements to the inference loop, release b9654 demonstrates llama.cpp's commitment to maintaining its status as the most universally compatible inference engine available. The Windows build matrix now explicitly includes support for CUDA 13, shipping with CUDA 13.3 DLLs alongside the existing CUDA 12.4 binaries.
This dual-support strategy is critical for enterprise deployments. While CUDA 12 remains the standard for the vast majority of current production environments utilizing Ampere and Ada Lovelace architectures, the transition to CUDA 13 is necessary to fully exploit the capabilities of Nvidia's Hopper and upcoming Blackwell architectures. By providing pre-built binaries for both, llama.cpp reduces the friction for teams migrating their local inference workloads to next-generation hardware. This ensures that optimizations specific to newer PTX (Parallel Thread Execution) instructions and memory management paradigms introduced in the CUDA 13 toolkit can be leveraged immediately by the community, without requiring complex local compilation chains.
Perhaps the most strategically important addition to the hardware matrix is the formalized support for openEuler, specifically targeting the Huawei Ascend ecosystem. The release notes detail specialized hardware targets for openEuler on both x86 and aarch64 architectures, explicitly supporting the 310p and 910b chips via the ACL (Ascend Computing Language) Graph backend.
The Ascend 910B is currently the most prominent domestic alternative to Nvidia hardware in the Chinese enterprise market, driven by ongoing geopolitical export controls. By integrating ACL Graph support, llama.cpp is not merely compiling for a new CPU architecture; it is interfacing directly with Huawei's proprietary neural processing unit (NPU) stack. The use of the Graph mode is particularly notable. Unlike single-operator execution, ACL Graph mode compiles the neural network operations into a static computational graph that is executed entirely on the NPU. This drastically reduces the overhead of CPU-to-NPU communication and maximizes the utilization of the Ascend hardware's matrix multiplication units. For global enterprises and developers operating in regions where Nvidia hardware is restricted or cost-prohibitive, native llama.cpp support for the 910B transforms the deployment landscape, offering a highly optimized, open-source alternative to vendor-locked inference servers.
Despite the forward momentum, the b9654 release presents several limitations and open questions that warrant closer examination. The most glaring regression is the explicit disabling of the macOS Apple Silicon build with KleidiAI enabled. KleidiAI is ARM's highly optimized microkernel library designed to accelerate AI workloads on ARM-based CPUs. Its integration into llama.cpp was intended to boost CPU inference performance on Apple Silicon.
The fact that this specific build target is marked as DISABLED in this release cycle suggests significant integration friction, potential memory leaks, or stability conflicts with Apple's own Accelerate framework or Metal Performance Shaders (MPS). Until the maintainers provide a detailed post-mortem, developers relying on CPU-bound inference on macOS should expect to fall back to standard ARM64 optimizations.
Furthermore, while the post-decode callback in the MTMD framework offers unprecedented control, the release lacks comprehensive documentation regarding its performance impact. Executing custom callback logic-especially if it involves complex Python bindings or heavy computational checks-during the tight inner loop of speculative decoding could introduce latency spikes that negate the performance benefits of drafting. The exact programmatic overhead of this callback remains an open question that will require rigorous benchmarking by the community.
The b9654 release of llama.cpp underscores a critical maturation phase for the project. It is evolving from a lightweight, CPU-focused inference engine into a highly modular, hardware-agnostic platform capable of orchestrating complex, multi-token generation pipelines. The introduction of post-decode callbacks provides the granular control necessary for enterprise-grade applications, while the aggressive expansion of the hardware matrix-spanning from the latest Nvidia CUDA 13 runtimes to Huawei's Ascend NPUs-ensures its relevance across a highly fragmented global hardware ecosystem. As local LLM deployment becomes increasingly heterogeneous, llama.cpp's strategy of universal compatibility and deep architectural extensibility solidifies its position as the foundational infrastructure for edge AI.
\n\n