lessw-blog explores the technical challenges of training linear classifiers to detect deceptive AI behavior with limited data.
\nIn a recent technical analysis, lessw-blog investigates the viability of "coup probes"—linear classifiers designed to monitor AI activations for signs of malicious intent or deceptive planning.
The Context: One of the central challenges in AI alignment is the potential for "scheming" models—systems that appear aligned on the surface while internally harboring non-compliant or hostile goals. To mitigate this, researchers are exploring methods to monitor the model's internal state (its "activations") rather than just its final output. However, a significant hurdle remains: training these monitors requires data. Because current models rarely exhibit genuine, unprompted malicious behavior (on-policy), researchers often rely on "off-policy" data generated by forcing the model to act badly. The discrepancy between these two data types creates a reliability gap in safety tools.
The Analysis: The post details an experiment aimed at quantifying the performance gains of coup probes when provided with a small number of on-policy examples—a "few-shot" approach to safety engineering. The methodology involves using Claude Sonnet 4.5 to generate trivia questions and Qwen2.5B-7B-Instruct to generate responses, creating a controlled environment to test how well a linear probe can distinguish between benign and potentially malicious internal states. The core inquiry focuses on whether a handful of genuine examples can bridge the gap left by synthetic training data.
This research is particularly significant for the "Risk - Safety" landscape. If linear probes can be effectively trained with minimal examples of bad behavior, it offers a scalable path toward monitoring future systems that may be capable of more sophisticated deception. The post serves as a technical exploration into the mechanics of activation monitoring and the specific difficulties of data scarcity in alignment research.
For those involved in mechanistic interpretability or AI safety, this analysis provides a concrete look at the trade-offs between off-policy and on-policy training data.
Read the full post on LessWrong
\n\n