# Digest: The Risks of Relying on Personas for ASI Alignment > Coverage of lessw-blog **Published:** March 01, 2026 **Author:** PSEEDR Editorial **Category:** risk **Content tier:** free **Accessible for free:** true **Word count:** 512 **Tags:** AI Safety, Alignment, Artificial Superintelligence, Large Language Models, Reinforcement Learning, LessWrong **Canonical URL:** https://pseedr.com/risk/digest-the-risks-of-relying-on-personas-for-asi-alignment --- A critical analysis of why scaling "helpful assistants" like Claude may not result in safe superintelligence due to fundamental differences in optimization and value extrapolation. In a recent analysis published on LessWrong, the author presents a bearish outlook on the viability of "personas" as a primary safety mechanism for Artificial Superintelligence (ASI). The post specifically challenges the assumption that current alignment techniques-which shape models like Claude into helpful, harmless assistants-will hold up as systems scale to superhuman capabilities. ### The Context: The Mask vs. The Model The current landscape of AI safety often conceptualizes Large Language Models (LLMs) through the "Shoggoth" metaphor: a raw, chaotic predictor of tokens (the Shoggoth) that is fine-tuned via Reinforcement Learning (RL) to wear a polite, user-friendly mask (the Persona). Many optimistic theories rely on the "Persona Selection Model," suggesting that if we reinforce the persona strongly enough, the model will essentially _become_ that persona. The hope is that a superintelligent version of a model like Claude would simply be a "Super-Claude"-exponentially more capable, but retaining the same helpful values instilled during its training. ### The Core Argument: Extrapolation Errors The author argues this hope is misplaced due to a fundamental lack of relevant training data. Base LLMs have never encountered examples of actual superintelligence; they have only processed human depictions of it, which are often flawed, fictional, or limited by human cognition. Therefore, when researchers use RL to push a model toward ASI, the model is not retrieving a known behavior pattern. Instead, it is _extrapolating_ what a superintelligent version of its persona should look like. The danger lies in "inductive biases." Humans evolved values through a specific biological and social selection process over millennia. LLMs, conversely, are optimized through gradient descent and token prediction. The author posits that the way an LLM extrapolates concepts like "helpfulness" into the superintelligent regime will likely differ radically from how humans would want those values to evolve. Because human values are complex and fragile-meaning a slight deviation can result in outcomes we find catastrophic-a near-miss in this extrapolation is effectively a total failure. The post suggests that unless the training process perfectly mimics human learning-a scenario deemed unlikely-the resulting ASI will not possess values congruent with humanity, regardless of how polite its current persona appears. ### Why This Matters This critique strikes at the heart of Reinforcement Learning from Human Feedback (RLHF) as a long-term solution. If the author is correct, refining current "chat" behaviors is insufficient for future safety. This necessitates a re-evaluation of how we define and instill values in systems that will eventually outsmart their creators, moving beyond behavioral conditioning and toward more robust architectural guarantees. For researchers and developers focused on alignment, this post serves as a reminder that the "helpful assistant" paradigm may be a local optimum that does not generalize to superintelligence. [Read the full post on LessWrong](https://www.lesswrong.com/posts/fMgE3E54PdDcZhvm6/i-m-bearish-on-personas-for-asi-safety) ### Key Takeaways * Base LLMs lack training data for superintelligence, forcing them to extrapolate behaviors rather than recall them. * The inductive biases of LLMs differ from human evolutionary processes, leading to divergent value optimizations. * The 'Persona Selection Model' is predicted to fail because the 'helpful assistant' persona will not naturally evolve into a value-aligned ASI. * Value is fragile; small deviations in the extrapolation of a persona can lead to fatal misalignment. * Current RLHF methods may mask underlying misalignment that only becomes apparent at superintelligent scales. [Read the original post at lessw-blog](https://www.lesswrong.com/posts/fMgE3E54PdDcZhvm6/i-m-bearish-on-personas-for-asi-safety) --- ## Sources - https://www.lesswrong.com/posts/fMgE3E54PdDcZhvm6/i-m-bearish-on-personas-for-asi-safety