PSEEDR

Llama.cpp Release b10051: Resolving Arm KleidiAI Kernel Dispatch for SME and SME2 Architectures

Granular runtime and build-time checks prevent instruction set mismatches on Arm-based LLM inference deployments.

· PSEEDR Editorial

The recent llama.cpp release b10051 introduces a critical fix for Arm KleidiAI integration, specifically correcting kernel dispatching logic for Scalable Matrix Extension (SME) and SME2 hardware. This update underscores the growing complexity of optimizing large language model (LLM) inference across fragmented Arm architectures, where failing to distinguish between instruction set revisions can lead to catastrophic execution failures on lower-tier hardware.

The SME Kernel Dispatch Vulnerability

The integration of Arm's KleidiAI into the llama.cpp ecosystem represents a significant push toward maximizing CPU-bound inference performance. However, release b10051 highlights a fundamental risk in hardware abstraction. Prior to this release, the llama.cpp codebase treated Arm's Scalable Matrix Extension (SME) as a monolithic capability, flagged under a single CPU_FEATURE_SME identifier. The underlying issue emerged because the kernels dispatched under this broad flag were compiled utilizing SME2-specific instructions.

For hardware equipped only with first-generation SME (SMEv1), this monolithic dispatch logic resulted in the execution of unsupported instructions. In C++ inference frameworks, instruction set architecture (ISA) mismatches do not degrade gracefully; they typically result in illegal instruction faults (SIGILL) and immediate process termination. This bug exposed a critical gap in how vendor-specific acceleration libraries are mapped to highly fragmented CPU microarchitectures, particularly as Arm rapidly iterates on its matrix math extensions to compete in the AI hardware space.

Architectural Implications for Arm Inference

The correction implemented in this release carries broader implications for developers targeting Arm-based infrastructure. As edge devices and cloud instances increasingly rely on Arm processors-ranging from mobile Cortex cores to server-grade Neoverse architectures-frameworks must implement highly granular hardware probing. KleidiAI is designed to provide highly optimized micro-kernels for AI workloads, but its efficacy is strictly bound by the host framework's ability to route workloads accurately.

By treating SME and SME2 as distinct execution paths, llama.cpp acknowledges the architectural divergence between the two extensions. SME2 introduces significant enhancements over SMEv1, including multi-vector instructions and improved data movement capabilities, which are highly advantageous for the memory-bound nature of LLM inference. Failing to distinguish between these extensions forces a compromise: either compile for the lowest common denominator (SMEv1), leaving SME2 performance on the table, or compile for SME2 and break compatibility with older hardware. The granular dispatching model ensures stability across the hardware spectrum while preserving the performance ceiling for modern processors.

Build-Time and Runtime Resolution Mechanics

Pull Request #25478, merged into release b10051, resolves the dispatching flaw by bifurcating the SME capability checks. The solution operates on two distinct layers: build-time configuration and runtime hardware detection.

At build time, the compiler toolchain is now instructed to generate separate object files or code paths for SME and SME2 kernels. This prevents the compiler from inadvertently injecting SME2 instructions into SMEv1-targeted binaries. At runtime, the framework queries the host CPU's feature registers to determine the exact level of SME support. The dispatching logic then dynamically wires the execution graph to the appropriate KleidiAI micro-kernels. This dual-layer approach guarantees that a single compiled binary can safely execute on SMEv1 hardware while automatically scaling up to utilize SME2 instructions when deployed on newer silicon. The release notes confirm that these targeted builds span multiple operating systems, including macOS Apple Silicon, Windows arm64, Linux arm64, and Android arm64.

Limitations and Unresolved Variables

While the dispatch logic correction stabilizes execution, several technical variables remain unaddressed in the current release documentation. First, the exact performance delta between the SMEv1 and SME2 KleidiAI kernels during LLM inference is not quantified. For infrastructure engineers provisioning Arm-based cloud instances, understanding the tokens-per-second throughput difference between these two instruction sets is critical for cost-to-performance optimization.

Furthermore, the release notes explicitly list the "macOS Apple Silicon (arm64, KleidiAI enabled)" build target as disabled. The documentation does not specify whether this is due to a temporary build pipeline failure, a compatibility issue with Apple's specific implementation of the Arm architecture (which relies heavily on its proprietary AMX coprocessor for matrix math rather than standard SME), or an unresolved bug within KleidiAI itself on Darwin systems. Finally, the exact CPU models affected by the SMEv1-only hardware dispatch bug-such as specific iterations of Arm Neoverse or Cortex cores-are omitted, leaving developers to manually cross-reference CPU feature sheets to determine their exposure to the original bug.

Synthesis

The b10051 release of llama.cpp illustrates the maturation process of open-source AI frameworks as they adapt to increasingly specialized hardware. The transition from a monolithic SME flag to granular SME and SME2 dispatching reflects the necessary evolution of runtime environments to support advanced, vendor-specific acceleration libraries like KleidiAI. As silicon vendors continue to iterate on matrix math extensions, the burden of stability falls on the framework's ability to accurately probe and route instructions. This update ensures that the pursuit of maximum inference throughput on cutting-edge Arm processors does not compromise the reliability of deployments on legacy or lower-tier hardware.

Key Takeaways

  • Llama.cpp release b10051 fixes a critical kernel dispatch bug that caused SME2 instructions to execute on SMEv1 hardware.
  • The update introduces distinct build-time and runtime checks to accurately route Arm KleidiAI micro-kernels based on host CPU capabilities.
  • Granular hardware probing prevents illegal instruction faults (SIGILL) on legacy Arm processors while maintaining peak performance on modern silicon.
  • The KleidiAI-enabled build for macOS Apple Silicon remains disabled in this release, leaving questions about Darwin compatibility unresolved.

Sources