The recent llama.cpp b9788 release introduces a custom SYCL tensor parallelism implementation specifically optimized for dual-GPU setups. By intentionally bypassing Intel's OneCCL collective communication library, the update highlights a growing trend where open-source inference engines deploy lightweight, custom-built primitives to maximize performance on consumer-grade and workstation hardware.
\nThe recent llama.cpp b9788 release introduces a custom SYCL tensor parallelism implementation specifically optimized for dual-GPU setups. By intentionally bypassing Intel's OneCCL collective communication library, the update highlights a growing trend where open-source inference engines deploy lightweight, custom-built primitives to maximize performance on consumer-grade and workstation hardware.
\n\nAt the core of this update is a fundamental architectural conflict between enterprise-grade communication libraries and the requirements of local, single-node inference. Intel's OneCCL (specifically version 2021.17) enforces a strict single-device-per-process constraint, hardcoded as devices.size() == 1 within its communicator_impl.hpp file. This design assumes a traditional distributed computing environment where each process manages exactly one accelerator, typically orchestrated via MPI.
However, llama.cpp operates on a single-process, multi-GPU model. Instead of refactoring the entire inference engine to accommodate Intel's enterprise-centric library, the developers opted to write a custom SYCL implementation. This custom trio of commands-comm_init, comm_free, and comm_allreduce_tensor-mirrors the pattern previously established by ggml-cuda.cu. By querying these commands via get_proc_address, the meta-backend can execute backend-specific all-reduce operations without relying on rigid vendor middleware. This demonstrates that vendor-supplied collective communication libraries are frequently too heavyweight for the low-latency demands of local Large Language Model (LLM) inference.
To maximize efficiency on dual-GPU (N=2) configurations, the release implements a degenerate ring all-reduce featuring a size-branched path. This design directly targets PCIe bandwidth, which is frequently the primary bottleneck in tensor parallelism.
\ndepends_on(memcpy_event). This path requires four SYCL submissions per call and is optimized for minimal overhead.Memory management is kept strictly efficient. A single persistent uint8_t buffer per device handles both paths, maintaining the SYCL memory pool's strict Last-In-First-Out (LIFO) invariant. Furthermore, per-call synchronization is intentionally omitted. The implementation relies on SYCL's in-order queue semantics to ensure that the meta-backend's subsequent compute operations wait for the final ADD operation, minimizing idle cycles.
The real-world impact of this custom implementation is substantial, particularly for memory-bound decoding phases. Benchmarks conducted on dual Intel Arc Pro B70 GPUs (using NEO 26.05.x and oneAPI 2025.3) show dramatic improvements over the standard layer-splitting mode.
\nRunning Llama-3.3-70B (Q4_K_M), the custom SYCL tensor parallelism achieved 17.40 tokens per second (t/s) in token generation (tg128), representing a 78.6% increase over the 9.74 t/s achieved in layer mode. Prefill performance (pp512) also saw a 20.2% boost, reaching 377.08 t/s. Smaller models also scale effectively, with Qwen3-4B (Q4_K_M) hitting 984.51 t/s in prefill and 49.29 t/s in generation.
\nThe benefits extend to Mixture of Experts (MoE) architectures. Qwen3-Coder-Next-80B-A3B (Q3_K_M) achieved 216.56 t/s in prefill, a 38.3% increase over the generic meta-backend butterfly fallback. The developer notes that the BF16 compression path is specifically responsible for facilitating the \"many-medium-allreduces\" pattern required by MoE models during the prefill phase.
\n\nWhile the b9788 release significantly improves dual-GPU performance, several technical limitations and open questions remain. First, the implementation is explicitly hardcoded for N=2 GPUs. Setups with three or more Intel GPUs will fall back to the generic meta-backend butterfly implementation, the exact mechanics and scaling efficiency of which remain undocumented in this release context.
\nSecond, the exact architectural or historical reasons why Intel's OneCCL maintains the strict single-device-per-process constraint are not detailed. Understanding whether this is a hard limitation of the underlying hardware telemetry or merely a legacy software design choice is critical for the long-term viability of Intel GPUs in local AI clusters.
\nFinally, the release lacks specific data regarding the PCIe generation and lane configuration of the Intel Arc Pro B70 testbed. Because the BF16 compression path is designed specifically to mitigate PCIe bottlenecks, the exact bandwidth threshold where the six-submission overhead outweighs the bandwidth savings will vary significantly depending on whether the host system utilizes PCIe Gen 4x16, Gen 4x8, or older standards.
\n\nThe introduction of custom SYCL tensor parallelism in llama.cpp represents a pragmatic shift in how open-source inference engines handle non-NVIDIA hardware. By identifying and bypassing the rigid constraints of vendor-supplied libraries like OneCCL, developers are extracting highly competitive performance from consumer and workstation-grade Intel GPUs. This approach not only lowers the barrier to running massive 70B+ parameter models on alternative hardware but also signals to hardware vendors that lightweight, flexible communication primitives are becoming just as critical as raw compute capability in the decentralized AI ecosystem.
\n\n