# Mechanistic Interpretability Exposes Half-Baked Circuits in Character-Level Transformers > An analysis of a custom transformer reveals a structural disconnect between a model's representational capacity and its routing execution. **Published:** July 10, 2026 **Author:** PSEEDR Editorial **Category:** platforms **Content tier:** free **Accessible for free:** true **Editorial format:** analysis **News quality eligible:** true **Source count:** 1 **Word count:** 1071 **Tags:** Mechanistic Interpretability, Transformer Architecture, AI Safety, Attention Circuits, Neural Networks **Canonical URL:** https://pseedr.com/platforms/mechanistic-interpretability-exposes-half-baked-circuits-in-character-level-tran --- Recent mechanistic interpretability research published on [lessw-blog](https://www.lesswrong.com/posts/x2ZdCoqAaLedpjDk3/an-induction-head-in-disguise-chasing-grammar-in-a-character) demonstrates how a character-level transformer can develop the output machinery for complex grammar without the routing capability to execute it. This structural disconnect between the Output-Value (OV) and Query-Key (QK) circuits highlights a critical risk in AI safety: the presence of incomplete circuits that possess latent representational capacity but fail in practical application. ## Deconstructing the Character-Level Architecture To isolate and observe the formation of localized grammar and copying behaviors, the researcher trained a custom character-level transformer exclusively on the texts of Friedrich Nietzsche. The architecture is relatively standard but scaled down for interpretability: a residual stream fed by six blocks, with each block containing four attention heads and a standard Multilayer Perceptron (MLP) utilizing a Linear-Activation-Linear structure. The residual stream acts as the central communication channel, passing the evolving token representations through the network, while the MLPs process these representations non-linearly. By operating at the character level, the model is forced to learn syntax, punctuation, and word boundaries from scratch, providing a pristine environment for observing the emergence of fundamental linguistic circuits. The analysis relies heavily on separating the attention heads into two distinct mechanisms: the Query-Key (QK) circuit and the Output-Value (OV) circuit. The QK circuit is responsible for routing-it determines what the head should attend to within the context window. The OV circuit is responsible for representation-it dictates which token representations should be boosted and written to the residual stream once the attention routing is established. To identify specific functional heads, the researcher applied the eigenvalue method proposed in Elhage et al.'s foundational paper, _A Mathematical Framework for Transformer Circuits_. By computing the eigenvalues for the OV circuit matrix across all blocks and heads, researchers can identify copying mechanisms. A positive eigenvalue indicates that passing an eigenvector through the OV matrix results in a scaled vector pointing in the same direction, effectively preserving and boosting the original token representation. ## Disconfirming Grammar and the Discovery of Incomplete Circuits Initial observations of the model's behavior led to hypotheses that specific attention heads were responsible for complex grammatical structures, such as quotation or parenthesis grammar. In mechanistic interpretability, it is common to observe a model successfully closing a bracket or a quote and assume a unified circuit is responsible for the entire operation. However, rigorous empirical testing disconfirmed these initial hypotheses. Instead of a complete grammatical circuit, the analysis revealed an induction head in disguise. The researcher identified a copying head that contained the precise OV circuit machinery required to execute the second half of a bracket-completion mechanism. The OV circuit knew exactly what token to boost to close the structure. However, the QK circuit was fundamentally insufficient. It lacked the capacity to route the attention required to actually trigger the bracket-completion mechanism in practice. This finding is highly significant because it demonstrates that a model can learn the representational logic for a task without learning the corresponding routing logic. The circuit is effectively half-baked. The model possesses the internal machinery to complete the bracket, but the control mechanism-the QK routing-breaks down, diverting the attention and preventing the conclusion of the grammatical operation. ## Implications for AI Safety and Interpretability The discovery of half-baked circuits introduces a severe complication for AI safety and mechanistic interpretability. Historically, interpretability efforts often focus on identifying specific capabilities by analyzing isolated components of a network. This case study serves as a strict warning against over-interpreting these isolated components. If researchers analyze only the OV circuit of a model, they might detect the latent capacity for a dangerous or highly complex behavior and falsely conclude that the model poses an immediate risk or possesses a specific capability. Conversely, if they observe the model failing to execute a task in practice due to a faulty QK circuit, they might falsely conclude the model is entirely incapable of that behavior. This structural disconnect means that latent capabilities can exist dormant within a model's weights. A model might possess the ammunition (the OV circuit's representational capacity) without the trigger (the QK circuit's routing capacity). From a security perspective, this is a vulnerability. Minor fine-tuning, weight perturbations, or even highly specific prompt injections could theoretically bypass the faulty QK routing, suddenly unlocking the latent capability stored in the OV circuit. Therefore, safety evaluations must assess the holistic integration of QK and OV circuits rather than treating the presence of representational capacity as synonymous with operational capability. ## Limitations and Missing Context While the analysis provides a compelling look at circuit disconnects, several technical limitations restrict the broader applicability of the findings. The source material lacks the specific mathematical criteria and eigenvalue thresholds used to definitively classify the head as a copying head. Without these thresholds, it is difficult to determine the boundary between a weak copying mechanism and a definitive functional circuit. Furthermore, the exact failure mode of the QK circuit remains undefined. It is unclear whether the routing failure is due to an attention sink issue, a breakdown in positional encoding integration, or simply undertrained weights that failed to converge on the correct routing logic during the Nietzsche training run. Finally, critical hyperparameters of the character-level transformer are missing from the documentation. The context window length, embedding dimensions, and learning rate schedules are essential for reproducing this specific state of half-baked circuit formation. Without this context, it is challenging to determine if this disconnect is a common artifact of character-level training or an anomaly specific to this exact architectural configuration. ## Synthesis The identification of an incomplete bracket-completion mechanism within a custom transformer underscores the fragility of circuit formation in neural networks. By proving that a model can develop robust Output-Value representations while failing to develop the necessary Query-Key routing, this analysis forces a reevaluation of how latent capabilities are measured. Mechanistic interpretability must move beyond the identification of isolated features and focus on the functional integration of routing and representation. Understanding how and why these structural disconnects occur is essential for predicting model behavior and preventing the sudden emergence of dormant capabilities. ### Key Takeaways * Mechanistic interpretability of a custom character-level transformer revealed a structural disconnect between representational and routing circuits. * The model developed an Output-Value (OV) circuit capable of bracket-completion, but lacked the Query-Key (QK) routing to execute it. * Latent capabilities can exist dormant within a model, posing risks if safety evaluations rely solely on isolated component analysis. * Missing hyperparameters and exact eigenvalue thresholds limit the immediate reproducibility of the specific QK failure mode. --- ## Sources - https://www.lesswrong.com/posts/x2ZdCoqAaLedpjDk3/an-induction-head-in-disguise-chasing-grammar-in-a-character