# The Hoarding Transformer: Why Models Retain Defunct State Data Absent Capacity Pressure > Mechanistic interpretability research reveals that toy transformers do not automatically prune obsolete predictive features, challenging strict optimal-prediction theories. **Published:** June 24, 2026 **Author:** PSEEDR Editorial **Category:** platforms **Content tier:** free **Accessible for free:** true **Editorial format:** analysis **News quality eligible:** true **Source count:** 1 **Word count:** 1026 **Tags:** Mechanistic Interpretability, Transformers, Model Compression, Optimal-Prediction Theory, Machine Learning **Canonical URL:** https://pseedr.com/platforms/the-hoarding-transformer-why-models-retain-defunct-state-data-absent-capacity-pr --- Recent mechanistic interpretability research published on [lessw-blog](https://www.lesswrong.com/posts/vzav5kfbRCDQyEB8v/toy-transformers-may-represent-belief-state-geometry) demonstrates that toy transformers retain predictively defunct belief-state data in their residual streams unless subjected to explicit capacity pressure. For PSEEDR, this finding highlights a critical architectural inertia: transformers act as representational hoarders rather than minimal optimizers, carrying significant implications for model compression, fine-tuning efficiency, and the development of internal world models. ## The Illusion of Optimal Prediction in Toy Models The prevailing assumption in many mechanistic interpretability frameworks is rooted in optimal-prediction theory. This theory posits that a neural network, driven by the need to minimize loss, will naturally adopt the most energetically and computationally inexpensive method to represent the information required for next-token prediction. In theory, a model should recognize when a feature is no longer relevant to its predictive task and prune that representation from its internal state to free up capacity. The research builds upon the foundation laid by Shai et al. (2024), which utilized Hidden Markov Models (HMMs) to train toy transformers on next-token prediction tasks. By analyzing the values within the transformer's residual stream, researchers can map out Bayesian belief state representations regarding the underlying HMM. Because referencing these state representations is more efficient than utilizing a brute-force lookup table for token strings, the optimal-prediction framework appeared to be a sound theoretical lens. To test this, the author introduced a stochastic token generation element-referred to as a "coin"-into the training environment. This coin meaningfully affected the state for a specific number of epochs before becoming entirely irrelevant to future token predictions. Under a strict interpretation of optimal-prediction theory, the transformer should have discarded the data regarding the coin once it became predictively defunct. However, the initial experiments disproved this hypothesis. The transformer failed to automatically prune the obsolete coin data from its residual stream. ## The Mechanics of Representational Hoarding The core finding of the experiment is that toy transformers do not default to minimal representations. Instead, they exhibit a phenomenon that can be described as representational hoarding. The predictively defunct belief-state data remains embedded within the residual stream, occupying representational space despite offering no ongoing utility for the next-token prediction objective. The research indicates that this defunct information is only shed when the model is subjected to sufficient "capacity pressure." When the model is forced to learn new, predictively vital information but lacks the residual stream bandwidth to store it alongside the hoarded data, it finally begins to overwrite the obsolete representations. Furthermore, the shedding process follows a specific chronological pattern: when capacity pressure is applied, the oldest predictively defunct information is discarded first. This behavior suggests that the loss landscape for transformers is relatively flat regarding the retention of useless information, provided the model has excess capacity. The network does not actively penalize the retention of defunct state variables; it merely prioritizes the acquisition of new predictive signals when space becomes a strict constraint. ## Implications for Model Compression and Fine-Tuning From a PSEEDR perspective, the realization that transformers are hoarders rather than minimal optimizers introduces significant friction into enterprise AI workflows, particularly concerning model compression, alignment, and fine-tuning. If models naturally retain obsolete historical state representations, standard fine-tuning processes may be less efficient than previously assumed. When a model is fine-tuned to unlearn a specific behavior or adapt to a new domain, the defunct representations of its prior training may not be overwritten unless the fine-tuning process introduces severe capacity pressure. This architectural inertia could explain why models often exhibit "behavioral relapse" or why seemingly unlearned biases can be recovered through adversarial prompting. The obsolete belief states are not gone; they are simply dormant within the residual stream. Furthermore, this impacts the development of model compression techniques like pruning and quantization. If a significant portion of a model's internal representation consists of predictively defunct data, current pruning algorithms might struggle to differentiate between features that are actively contributing to the model's world state and features that are merely historical artifacts. Developing more efficient training objectives that artificially induce capacity pressure early in the training lifecycle could force models to maintain leaner, more minimal representations, ultimately reducing inference costs and improving the efficacy of downstream compression. ## Methodological Limitations and Open Questions While the findings present a compelling challenge to optimal-prediction theory, several methodological limitations and missing contextual elements require further investigation. The source text does not detail the specific mathematical or architectural mechanisms used to impose "capacity pressure" during the experiments. Understanding whether this pressure was induced by reducing the embedding dimension, increasing the complexity of the HMM, or altering the learning rate is crucial for replicating the shedding behavior. Additionally, the exact methodology for mapping and extracting Bayesian belief state representations from the residual stream remains unspecified in the provided brief. The reliability of these extraction techniques is paramount, as artifacts in the probing methodology could theoretically be misidentified as retained state data. Finally, there is the question of scale. These experiments were conducted on toy transformers trained on highly controlled Hidden Markov Models. Whether this representational hoarding behavior scales linearly to massive Large Language Models (LLMs) trained on diverse, unstructured internet text is an open question. High-parameter LLMs operate in regimes where capacity pressure is theoretically constant due to the vastness of the training data, yet they still exhibit memorization and feature redundancy. The retention of defunct belief-state data in toy transformers underscores a fundamental reality of neural network architecture: optimization for next-token prediction does not inherently equal optimization for minimal internal representation. Recognizing that models require explicit capacity constraints to discard obsolete features forces a reevaluation of how training environments are designed. As the industry pushes toward more efficient and aligned systems, engineering the loss landscape to actively penalize representational hoarding may become a necessary step in advancing mechanistic interpretability and model performance. ### Key Takeaways * Toy transformers retain predictively defunct belief-state data in their residual streams, disproving strict optimal-prediction theory. * Obsolete predictive information is only shed when the model is subjected to sufficient capacity pressure. * When capacity pressure forces the model to overwrite data, the oldest predictively defunct information is discarded first. * The tendency of models to hoard historical state representations introduces friction for fine-tuning, alignment, and model compression techniques. --- ## Sources - https://www.lesswrong.com/posts/vzav5kfbRCDQyEB8v/toy-transformers-may-represent-belief-state-geometry