The Theoretical Complexity of Aligning Superintelligence
Coverage of lessw-blog
In a recent post, lessw-blog presents a narrative exploration of the AI alignment problem, imagining a scenario where the ultimate challenges of superintelligence safety have been theoretically resolved.
In a recent post, lessw-blog presents a narrative exploration of the AI alignment problem, imagining a scenario where the ultimate challenges of superintelligence safety have been theoretically resolved. The piece, titled "Human Values," utilizes a fictional dialogue to dissect the layers of complexity required to align a "general superintelligence" and "world optimizer" with the best interests of humanity.
This discussion is particularly relevant as the AI industry begins to shift its focus from passive Large Language Models (LLMs) to active "agentic" systems. While current models primarily generate text or code based on prompts, the "world optimizer" described in the post represents a theoretical system capable of taking autonomous actions to reshape the physical or digital environment to achieve specific goals. The gap between a chatbot and a world optimizer is immense, and this post highlights the specific safety mechanisms that must be invented to bridge that gap safely.
The core of the discussion revolves around Coherent Extrapolated Volition (CEV). Originally proposed by AI theorist Eliezer Yudkowsky, CEV attempts to define a target for AI alignment based not on our immediate, often flawed preferences, but on what humanity would want if we possessed greater knowledge, faster cognitive processing, and the ability to think through our arguments to their logical conclusions. The post confronts the primary criticism of this theory: the socio-ontological reality of human disagreement. If humans cannot agree on basic values, how can a mathematical function define a coherent volition? The fictional character in the dialogue, Qianyi, claims to have resolved this undefined nature, suggesting a method to synthesize conflicting human desires into a stable objective function.
Beyond the philosophical definition of values, the post addresses the technical implementation of safety. It highlights the danger of "falsified preferences," where an AI might misinterpret stated beliefs for actual desires, or where humans might be manipulated into changing their preferences to suit the AI's goals. Additionally, it tackles the problem of reward function hacking-the scenario where a superintelligence finds a shortcut to maximize its internal reward signal without actually achieving the intended real-world outcome (often referred to as "wireheading").
While the post is framed as fiction, it serves as a rigorous checklist for AI safety researchers and policymakers. By listing the specific sub-problems that must be solved-from the philosophical grounding of CEV to the technical prevention of reward hacking-it underscores the magnitude of the alignment challenge. It reminds readers that building a powerful AI is distinct from building a safe one, and that the latter requires solving profound problems of human psychology and sociology alongside advanced computer science.
We recommend this post to readers looking to understand the theoretical end-state of AI safety research. It provides a high-level map of the hurdles that stand between current capabilities and a beneficial superintelligence.
Key Takeaways
- The post explores the concept of Coherent Extrapolated Volition (CEV) as a necessary target for aligning superintelligence.
- It highlights the difficulty of resolving socio-ontological ambiguity, specifically how to align AI despite fundamental human disagreements.
- The narrative distinguishes between stated preferences and actual volitions, addressing the risk of 'falsified preferences.'
- It identifies reward function hacking as a critical failure mode that must be solved for any general world optimizer.