# The HBM Wall: How Memory Bandwidth and Data Scarcity Will Constrain LLM Scaling Through 2031

> A hardware-grounded analysis reveals why extreme MoE sparsity and pipeline latency limits will dictate the next decade of frontier AI infrastructure.

**Published:** June 22, 2026
**Author:** PSEEDR Editorial
**Category:** stack
**Content tier:** free
**Accessible for free:** true
**Editorial format:** analysis
**News quality eligible:** true
**Source count:** 1
**Word count:** 1108


**Tags:** AI Infrastructure, Hardware Scaling, High Bandwidth Memory (HBM), LLM Architecture, Mixture of Experts (MoE)

**Canonical URL:** https://pseedr.com/stack/the-hbm-wall-how-memory-bandwidth-and-data-scarcity-will-constrain-llm-scaling-t

---

A recent analysis published on [LessWrong](https://www.lesswrong.com/posts/yLHiQGCPdvzL9fBn3/model-size-scaling-in-2023-2031) provides a rigorous, hardware-grounded roadmap for large language model (LLM) scaling through 2031. For enterprise AI infrastructure planners, the critical takeaway is a fundamental paradigm shift: raw compute (FLOPs) is no longer the primary bottleneck, as practical scaling will soon be strictly governed by High Bandwidth Memory (HBM) read speeds, pipeline latency, and an impending exhaustion of unique pretraining data.

## The Memory Bandwidth Wall: Why HBM Dictates Pipeline Depth

The conventional wisdom of AI scaling assumes that adding more compute linearly translates to larger, more capable models. However, token generation speed is fundamentally constrained by the physical time required to read weights and the Key-Value (KV) cache from High Bandwidth Memory (HBM). According to the source analysis, reading a full HBM stack takes 20 milliseconds on an NVIDIA H100, 30 milliseconds on an H200, and 24 milliseconds on a GB200. Future architectures like Rubin and the rumored Feynman chips are projected to reduce this to 13 milliseconds and 14-16 milliseconds, respectively.

When targeting a commercial serving speed of 80 tokens per second per request-even assuming a 3x speedup from speculative decoding or Multi-Token Prediction (MTP)-the system must generate a token every 37.5 milliseconds. This hard latency ceiling severely limits pipeline parallelism. For example, when using GB200 Oberon racks, reading the KV cache consumes approximately 12 milliseconds, leaving only 25.5 milliseconds for weight reads. Because each pipeline stage requires 12 milliseconds for weight reads, the maximum viable pipeline depth is strictly capped at two stages.

This pipelining constraint dictates the maximum feasible model size that can be served at acceptable latencies. Under these physical limits, the maximum FP4 parameter count scales from 1.3 trillion for H100 systems (2023) to 27 trillion for GB200 Oberon (2025), 442 trillion for Rubin Ultra Kyber (2028), and 2.9 quadrillion for Feynman (2030). Models exceeding these sizes on their respective hardware generations will fail to meet the 80 tokens-per-second threshold, regardless of the total FLOPs available in the cluster.

## The 2027 Data Scarcity Inflection Point

While memory bandwidth constrains inference, data scarcity will soon constrain pretraining. The total volume of unique, high-quality pretraining data is projected to cap at approximately 200 trillion tokens. Meanwhile, pretraining compute budgets are scaling exponentially, moving from 2.1e25 FLOPs in 2023 (roughly 20,000 A100s) to a projected 1.3e27 FLOPs in 2026 (200,000 H100s), and potentially reaching 2.2e29 FLOPs by 2031, powered by clusters drawing up to 40 gigawatts of IT power.

Starting in 2027, the industry will hit a critical inflection point: compute availability will vastly outstrip the supply of unique training data. To absorb this massive influx of compute without severely degrading training efficiency through excessive data repetition, models will be forced to scale up their parameter counts far beyond what current compute-optimal scaling laws (such as Chinchilla) would predict. By 2031, models will need to be up to 4x larger than unlimited-data scaling laws dictate simply to utilize the available pretraining FLOPs effectively.

## Extreme MoE Sparsity as the Survival Mechanism

To reconcile the opposing forces of data-driven parameter inflation and HBM-constrained inference latency, architectures must aggressively transition toward extreme Mixture-of-Experts (MoE) sparsity. Higher sparsity allows a model to increase its total parameter count-absorbing excess compute during pretraining-without proportionally increasing the number of active parameters that must be read from HBM during inference.

The analysis projects a shift from 8x sparsity in 2026 (supporting 10 trillion parameter models constrained by Oberon racks) to 30x sparsity by 2028. This extreme 30x sparsity will enable models to reach 240 trillion parameters on Kyber racks, eventually scaling to 1.4 quadrillion parameters by 2031 on Feynman architecture. Without this architectural pivot, the gap between available pretraining compute and inference memory bandwidth would render future frontier models economically unviable to serve.

## Strategic Implications for AI Infrastructure

For organizations investing in AI infrastructure, this analysis underscores a critical transition: the industry is moving from a compute-bound era to a memory-bandwidth-bound and data-bound era. Raw FLOPs are diminishing in value as a standalone metric. Instead, the limiting factors for frontier AI development will be HBM capacity, HBM read speeds, and the network topology within scale-up systems (such as NVIDIA's NVLink domains).

Infrastructure planning must prioritize hardware-software co-design. Because expert parallelism must increasingly occur within scale-up systems rather than across them to avoid prohibitive networking overhead, the physical design of server racks-specifically the ratio of compute dies to HBM stacks-will dictate the upper bounds of model architecture. Enterprises must evaluate future hardware acquisitions not just on theoretical compute throughput, but on their ability to support hyper-sparse MoE routing within strict latency budgets.

## Limitations and Open Questions

While the scaling projections provide a robust framework, they rely on several unverified assumptions regarding future hardware and software. The exact architectural differences between NVIDIA's Oberon and Kyber rack designs remain unspecified, and the hardware specifications for the rumored Feynman chip architecture are speculative. Furthermore, the model assumes that Multi-Token Prediction (MTP) or speculative decoding will reliably deliver a 3x speedup in production environments, a software optimization that is highly dependent on workload characteristics.

Additionally, the analysis does not fully quantify the impact of Reinforcement Learning with Verifiable Rewards (RLVR) or extended inference-time reasoning (such as OpenAI's o1 architecture) on parameter scaling. If inference compute can be substituted for pretraining compute through advanced reasoning techniques, the pressure to inflate parameter counts in response to data scarcity may be partially mitigated.

## Synthesis

The trajectory of LLM scaling over the next decade will not be defined by the brute-force accumulation of compute, but by the delicate balancing of physical memory constraints and finite training data. As the industry approaches the limits of HBM read speeds and exhausts the supply of unique human-generated text, extreme MoE sparsity and rigorous pipeline management will become mandatory. Navigating this transition requires a fundamental reassessment of how AI infrastructure is evaluated, shifting the focus from theoretical FLOPs to the practical realities of memory bandwidth and latency.

### Key Takeaways

*   Token generation speed is physically capped by HBM read times, strictly limiting pipeline depth to maintain target latencies (e.g., 80 tokens/second).
*   A projected cap of 200 trillion unique pretraining tokens will force models to inflate parameter counts up to 4x beyond compute-optimal laws starting in 2027.
*   To balance parameter inflation with inference latency limits, models will transition to extreme MoE architectures, reaching up to 30x sparsity by 2028.
*   Hardware constraints dictate that total FP4 parameters will scale from 1.3T on H100 systems to 2.9 quadrillion on projected 2030 Feynman architectures.
*   AI infrastructure planning must shift focus from raw FLOPs to HBM capacity, read speeds, and intra-rack network topology.

---

## Sources

- https://www.lesswrong.com/posts/yLHiQGCPdvzL9fBn3/model-size-scaling-in-2023-2031