Beyond Linear Probing: Uncovering Magnitude-Encoded Features in Activation Space

In a recent walkthrough of a BlueDot Technical AI Safety puzzle, a researcher on lessw-blog demonstrated a critical blind spot in standard mechanistic interpretability techniques. The analysis reveals that a small text classifier packed two independent features onto a single activation axis using both sign and magnitude, a finding that directly challenges the Linear Representation Hypothesis and highlights the necessity of non-linear probing for AI safety.

The standard paradigm in mechanistic interpretability heavily relies on the Linear Representation Hypothesis-the assumption that neural networks represent high-level concepts as linear directions in activation space. However, a recent analysis of a BlueDot Technical AI Safety puzzle published on lessw-blog exposes a concrete failure mode of this assumption. By examining a small text classifier, the author discovered that the model successfully packed two completely independent features onto a single activation direction. This was achieved by encoding one feature via the sign of the projection and the other via its magnitude, rendering standard linear probes entirely blind to the second feature.

Architecture and the BlueDot Puzzle Context

The model under investigation is a relatively compact text classifier. It utilizes an all-MiniLM-L6-v2 encoder, which applies mean-pooling to convert input text into a 384-dimensional vector. This vector is then passed through a 5-layer Multilayer Perceptron (MLP) equipped with ReLU activations, culminating in eight independent sigmoid outputs. These outputs correspond to eight simple, surface-level binary features: number, question, color, food, sentiment, country, person, and body_part.

Crucially, these features are not mutually exclusive. A single input text, such as a query asking about Italian cuisine, can simultaneously trigger both the "question" and "food" features. Consequently, the model outputs independent probabilities rather than a mutually exclusive softmax distribution. Despite its small size, the model achieves over 95 percent accuracy across all eight independent binary features. The interpretability challenge focused on analyzing the 64-dimensional activation space located immediately after the ReLU activation in the second hidden layer.

The Sign-Magnitude Packing Mechanism

When researchers search for features in activation space, they typically look for distinct vectors. If a model needs to represent more features than it has dimensions, it often relies on superposition, where features are represented by almost-orthogonal vectors scattered throughout the high-dimensional space. The discovery in this 64-dimensional space, however, deviates from standard superposition models.

The analysis revealed that the model mapped two specific features-"food" and "country"-onto the exact same one-dimensional axis. Instead of using two slightly offset vectors, the network utilized the scalar properties of the activation value itself. The first feature is determined by the sign of the projection (whether the activation value is positive or negative along that axis). The second feature is determined by the magnitude of the projection (the absolute distance from the origin). This highly efficient packing mechanism allows the network to compress two independent binary variables into a single continuous dimension without interference, provided the downstream layers can decode this non-linear relationship.

The Blind Spot of Linear Probes

This sign-magnitude packing presents a severe problem for linear probes, which are the standard tool for extracting concepts from hidden layers. A linear probe operates by finding a hyperplane that separates the activation space into two regions-essentially checking if a projection along a specific vector is greater than a learned threshold.

In this scenario, a linear probe can easily detect the feature encoded by the sign, as it simply requires a hyperplane passing near the origin perpendicular to the axis. However, the linear probe is mathematically incapable of detecting the magnitude-encoded feature. To isolate a feature based on magnitude, a classifier must define a bounded interval (e.g., identifying activations that fall between -2 and 2, or outside that range). A single linear hyperplane cannot carve out an enclosed interval on a 1D line. As a result, the linear probe registers the magnitude-encoded feature as noise, completely missing a highly accurate, independent concept actively used by the model. To bypass this limitation, the author implemented a second-order boundary algorithm-effectively a non-linear or quadratic probe-capable of detecting the magnitude thresholds. The code demonstrating this approach has been published in the author's GitHub repository.

Implications for AI Safety and Interpretability

The implications of this finding extend far beyond a single interpretability puzzle. In the context of AI safety, the Linear Representation Hypothesis is not just an academic curiosity; it is the foundation upon which many safety monitors, concept erasure techniques, and alignment interventions are built. If safety researchers assume that dangerous concepts (such as deception, toxicity, or bioweapons knowledge) will always manifest as linear directions, they will rely exclusively on linear probes to audit models.

This analysis proves that models possess the capacity to hide critical concepts in non-linear structures along already-utilized activation directions. If a model learns to encode a benign feature via the sign of an axis and a safety-critical feature via its magnitude, standard linear audits will certify the model as safe while the dangerous capability remains fully intact and operational. Addressing this vulnerability requires a paradigm shift toward non-linear probing techniques, such as the second-order boundaries demonstrated in this puzzle. However, adopting non-linear probes introduces significant trade-offs. It dramatically increases the computational overhead of auditing large language models and raises the risk of overfitting, where a highly expressive non-linear probe might "discover" features in random noise rather than identifying true, causally relevant representations.

Limitations and Open Questions

While the demonstration of sign-magnitude packing is compelling, several limitations and open questions remain regarding its broader applicability. The most significant unknown is whether this multi-feature packing is an emergent property of standard stochastic gradient descent (SGD) or if it was explicitly engineered into the model weights by the creators of the BlueDot puzzle. If it is an artifact of intentional puzzle design, it serves as a valuable theoretical counterexample but may not reflect the natural behavior of models trained on massive, unstructured datasets. If it is emergent, it suggests that neural networks routinely exploit non-linear scalar properties for compression, which would fundamentally alter how we approach model auditing.

Furthermore, the source material leaves the exact mathematical formulation of the applied second-order boundary somewhat abstracted, requiring researchers to consult the accompanying codebase for implementation details. It is also unclear how the remaining six features are distributed across the other 63 dimensions of the activation space, and whether they exhibit similar non-linear packing or adhere to standard orthogonal representations.

The discovery that a neural network can effectively hide an independent feature within the magnitude of an already occupied activation axis forces a reevaluation of current interpretability toolkits. Relying solely on linear probes is insufficient for rigorous AI safety audits, as models can and do utilize non-linear scalar properties to compress information. Moving forward, the interpretability community must balance the necessity of non-linear detection methods with the computational costs and overfitting risks they introduce, ensuring that hidden capabilities cannot evade detection simply by changing their geometric shape in activation space.

Key Takeaways

• A small text classifier successfully packed two independent features ('food' and 'country') onto a single activation axis in a 64-dimensional space.
• The model achieved this by encoding one feature using the sign of the projection and the second feature using the magnitude, allowing both to coexist without interference.
• Standard linear probes are mathematically blind to magnitude-encoded features, highlighting a critical failure mode in current mechanistic interpretability techniques.
• Detecting these hidden features requires non-linear techniques, such as second-order boundaries, which complicates AI safety audits and challenges the Linear Representation Hypothesis.