# llama.cpp b9777 Integrates Liquid AI LFM2.5 Models for Edge-Native RAG > The addition of LFM2.5-ColBERT-350M and LFM2.5-Embedding-350M signals a shift toward non-Transformer architectures for local retrieval pipelines. **Published:** June 24, 2026 **Author:** PSEEDR Editorial **Category:** edge **Content tier:** free **Accessible for free:** true **Editorial format:** analysis **News quality eligible:** true **Source count:** 1 **Word count:** 924 **Tags:** llama.cpp, Liquid AI, RAG, ColBERT, Edge AI, Machine Learning **Canonical URL:** https://pseedr.com/edge/llamacpp-b9777-integrates-liquid-ai-lfm25-models-for-edge-native-rag --- The recent release of [llama.cpp b9777](https://github.com/ggml-org/llama.cpp/releases/tag/b9777) introduces native support for Liquid AI's LFM2.5-ColBERT-350M and LFM2.5-Embedding-350M models. By integrating these non-Transformer architectures into its model registry, the project establishes a foundation for executing high-performance, low-latency Retrieval-Augmented Generation (RAG) pipelines directly on edge devices and consumer hardware. ## Expanding the Local RAG Ecosystem The integration of Liquid AI's LFM2.5 models into llama.cpp via PR #24913 marks a critical expansion of the framework's capabilities. Historically, the llama.cpp project has focused predominantly on optimizing the inference of large language models (LLMs) for generative tasks. While basic embedding support has existed for some time, the addition of LFM2.5-ColBERT-350M and LFM2.5-Embedding-350M introduces specialized, state-of-the-art retrieval architectures directly into the local execution environment. Liquid Foundation Models (LFMs) utilize a dynamic, non-Transformer architecture designed for high computational efficiency and significantly reduced memory footprints compared to traditional attention mechanisms. By bringing these 350-million-parameter models into the llama.cpp registry, developers can now construct complete Retrieval-Augmented Generation (RAG) pipelines that operate entirely on-device. This shift allows applications to bypass the latency, cost, and privacy concerns associated with relying on cloud-based embedding or search APIs, keeping sensitive enterprise or personal data strictly local. ## Hardware Ubiquity and Backend Optimization The b9777 release notes highlight an aggressive cross-platform compilation strategy, ensuring these new retrieval models can be deployed across a vast array of consumer and edge hardware. The release assets include specialized builds for macOS Apple Silicon, notably featuring KleidiAI enablement. KleidiAI provides highly optimized micro-kernels for ARM architectures, which is crucial for maximizing throughput on mobile and edge devices. Windows support extends beyond standard x64 CPUs to include Windows arm64 with OpenCL Adreno support, catering directly to the growing market of Snapdragon-powered AI PCs and ensuring that local search does not drain battery life unnecessarily. Furthermore, the inclusion of openEuler targets with Ascend 910b and 310p ACL Graph support indicates a strong push into enterprise and specialized NPU environments. On the GPU front, the release maintains comprehensive backend compatibility, supporting CUDA 12.4 and 13.3, Vulkan, ROCm 7.2, OpenVINO, and SYCL. This broad hardware support ensures that the LFM2.5 retrieval models can be hardware-accelerated regardless of the underlying host architecture, a critical requirement for latency-sensitive search operations where time-to-first-token is paramount. ## Implications for Late-Interaction Retrieval The inclusion of a ColBERT-style model is particularly noteworthy for the future of local search. Traditional embedding models compress entire documents or chunks into single dense vectors. While efficient for vector databases, this approach can lead to significant information loss, especially for complex or nuanced queries. ColBERT (Contextualized Late Interaction over BERT) solves this by preserving token-level embeddings for both the query and the document, computing similarity scores through a late interaction phase. This significantly improves retrieval accuracy but is traditionally computationally intensive and memory-heavy, making it difficult to run on edge devices. Liquid AI's application of the LFM architecture to the ColBERT paradigm at a compact 350M parameter scale suggests a highly optimized approach to this problem. Running this within llama.cpp means developers can leverage the framework's advanced GGUF quantization techniques. By quantizing the token embeddings to 8-bit or even 4-bit precision, the massive memory footprint typically associated with late-interaction search can be drastically reduced. This makes high-fidelity, late-interaction search feasible on devices with constrained RAM, such as smartphones, embedded systems, or entry-level laptops, fundamentally altering the economics of local search. ## Technical Limitations and Open Questions Despite the structural advantages of this integration, several technical unknowns remain unaddressed in the release documentation. The primary open question revolves around the performance and latency characteristics of LFM2.5-ColBERT-350M compared to traditional BERT-based ColBERT implementations (such as ColBERTv2) when running on edge hardware. While LFMs are designed for efficiency, the specific throughput of their late-interaction tensor operations within the llama.cpp backend requires independent benchmarking. ColBERT requires specialized MaxSim operations across token matrices; it is not yet clear how optimally llama.cpp's current tensor library (ggml) handles these specific computational patterns compared to standard attention blocks. Furthermore, the underlying architectural differences between LFMs and Transformers dictate different memory access patterns. While the README.md has been updated to restore LFM2 reference guidelines, comprehensive documentation regarding the optimal quantization strategies for LFM-based embeddings is still developing. Precision loss during quantization can disproportionately affect retrieval accuracy in late-interaction models, and the community will need to establish best practices for balancing GGUF compression ratios with search recall metrics. ## Synthesis The b9777 release of llama.cpp signifies a strategic maturation of the local AI software stack. By incorporating Liquid AI's LFM2.5 ColBERT and embedding models, the project moves beyond its origins as a pure inference engine for generative text, positioning itself as a comprehensive runtime for end-to-end AI applications. This development lowers the barrier for deploying sophisticated, low-latency search capabilities on consumer hardware, ensuring that the retrieval components of local RAG pipelines can operate with the same efficiency and privacy guarantees as the generative models they support. As the ecosystem digests these new non-Transformer architectures, the focus will inevitably shift toward benchmarking their real-world retrieval performance against established baselines, potentially redefining the standard for edge-native AI search. ### Key Takeaways * PR #24913 adds native support for Liquid AI's LFM2.5 ColBERT and Embedding 350M models to llama.cpp. * The release includes extensive cross-platform compilation targets, including ARM-optimized KleidiAI for macOS and OpenCL Adreno for Windows. * Running ColBERT architectures locally via llama.cpp enables advanced late-interaction retrieval on memory-constrained edge devices. * Performance benchmarks comparing LFM2.5-ColBERT to traditional BERT-based implementations on edge hardware remain an open question. --- ## Sources - https://github.com/ggml-org/llama.cpp/releases/tag/b9777