Analyzing Inkling: How Thinking Machines Engineered a 1-Trillion Parameter Open Multimodal MoE

The recent release of Inkling by Thinking Machines, detailed in a Hugging Face blog post, marks a significant milestone in open-weights AI: a 1-trillion parameter multimodal Mixture-of-Experts (MoE) model. By leveraging hybrid attention, short 1D convolutions (SConv), and shared-expert routing, Inkling directly addresses the compute and memory bottlenecks that typically restrict frontier-class models to proprietary APIs, signaling a shift in the viability of local, enterprise-grade deployment.

Architectural Innovations for Trillion-Parameter Scale

Scaling a model to nearly one trillion parameters (975B total) while maintaining inference efficiency requires aggressive architectural optimization. Inkling achieves this through a highly sparse Mixture-of-Experts (MoE) design, activating only 41 billion parameters during any given forward pass. The model utilizes a total of 256 experts, employing a routing mechanism that selects the top six active experts per token. Crucially, Inkling incorporates two always-active shared experts. This shared-expert sink handles generalized representations and common knowledge, preventing redundant computation across the routed experts and stabilizing the highly sparse routing process.

Beyond the MoE routing, Inkling departs from standard transformer conventions by replacing Rotary Positional Embeddings (RoPE) with a relative attention mechanism. Instead of injecting positional information into the embeddings prior to attention, Inkling learns position directly within the attention logits. A fourth projection generates a per-token, per-head relative feature, which is modified by the distance between the key and query vectors. To further optimize local feature extraction, the architecture introduces short 1D convolutions (SConv) over the hidden states. By reading the current token alongside the previous hidden states within a defined window, SConv effectively manages local token relationships. This offloads the burden of local representation from the primary attention and MoE modules, allowing them to dedicate parameters to complex, long-range reasoning.

Managing the 1-Million Token Context Window

Supporting a 1-million token context window natively is computationally prohibitive using standard global attention, which scales quadratically with sequence length. Inkling mitigates this memory bottleneck through a hybrid attention scheme. The decoder layers alternate between sliding window attention-which restricts the receptive field to a fixed, localized context-and full global attention.

The model employs a strict 5:1 ratio of sliding window layers to global attention layers. This configuration drastically reduces the Key-Value (KV) cache memory footprint and compute requirements for ultra-long sequences, while ensuring that information can still propagate across the entire 1-million token span during the global attention passes. The final layer of the network enforces global attention to synthesize feature-rich representations before generating the output logits. This hybrid approach is central to making a 1M context window viable on current hardware infrastructures.

Native Multimodality Without Complex Encoders

Unlike early multimodal systems that bolted massive, pre-trained vision or audio encoders (such as CLIP or Whisper) onto a text-only LLM, Inkling is a native, decoder-only multimodal model. It processes text, audio, and images through relatively lightweight embedding towers before passing them into the core transformer.

For vision, Inkling uses a hierarchical MLP (hMLP) patchifier. Linear layers progressively merge pixels until the final layer outputs a single embedding per patch. For audio, the model eschews complex waveform encoders in favor of discretized mel spectrograms. Audio is segmented into 100-millisecond chunks, converted to the mel scale, and classified into exact mel spectrogram bins. These discrete bin values are embedded and summed to construct the final audio input. Furthermore, image inputs include an additional temporal dimension to support video processing. By keeping the modality-specific towers simple, Inkling forces the massive 1T-parameter MoE backbone to handle the cross-modal reasoning, rather than relying on heavy preprocessing encoders.

Implications for Open-Source Deployment and Hardware

The release of a 1T-parameter model fundamentally challenges the open-source deployment ecosystem, primarily due to VRAM constraints. In its unquantized BF16 format, Inkling requires 2 terabytes of VRAM-necessitating a cluster of roughly twenty-five 80GB GPUs just to load the weights. However, Thinking Machines and the open-source community have aggressively targeted quantization to make deployment feasible.

The model includes a well-calibrated NVFP4 (Nvidia 4-bit Floating Point) variant, which reduces the VRAM requirement to 600 GB, making it deployable on a single 8-GPU node of next-generation Blackwell hardware or current high-density Hopper servers. Furthermore, day-zero integration with frameworks like vLLM, SGLang, and llama.cpp ensures immediate production readiness. Most notably, the community has already produced a 1-bit quantization via Unsloth, reducing the memory footprint by 95%. This proves that frontier-class, trillion-parameter models can be compressed and distributed for localized enterprise use, breaking the monopoly of API-only access for models of this scale.

Limitations and Open Questions

Despite the technical achievements, several critical details remain absent from the initial release data. Inkling was trained on a massive corpus of 45 trillion tokens, but the exact composition and distribution of this dataset across text, images, audio, and video modalities is undisclosed. Without this data mixture, it is difficult to assess the model's inherent biases or its true zero-shot capabilities across different domains.

Additionally, while the native audio and video processing architectures are elegant, there are no provided benchmarks comparing Inkling's performance against state-of-the-art specialized models. The effectiveness of its out-of-the-box video processing remains entirely unproven. Finally, the source mentions the inclusion of speculative Multi-Token Prediction (MTP) layers designed to accelerate inference, but lacks concrete architectural details or exact speedup metrics, leaving the practical impact of this feature ambiguous.

Ultimately, Inkling represents a structural blueprint for the next generation of open-weights AI. By combining sparse MoE routing, hybrid attention, and native multimodal tokenization, it demonstrates that the barriers to running trillion-parameter models are shifting from fundamental architectural limits to hardware and quantization engineering. As inference engines continue to optimize for formats like NVFP4 and 1-bit quantization, models of Inkling's scale will increasingly become viable for localized, enterprise-grade deployment.

Key Takeaways

• Inkling is a 1-trillion parameter open-weights multimodal MoE model that natively processes text, image, and audio inputs.
• The architecture achieves high inference efficiency by activating only 41 billion parameters out of 975 billion total during a forward pass.
• A hybrid attention mechanism, utilizing a 5:1 ratio of sliding window to global attention, enables a 1-million token context window.
• The model replaces traditional RoPE with relative attention and uses short 1D convolutions (SConv) to optimize local representations.
• Aggressive quantization, including an NVFP4 variant and a 1-bit Unsloth version, reduces the VRAM requirement from 2 TB to as low as 600 GB or less, enabling local deployment.