In a recent post on LessWrong, the author critiques the assumption that if AI alignment is a tractable engineering problem, humanity is automatically safe, arguing that even \"solvable\" problems can lead to catastrophe if mishandled.
\nIn a recent analysis, lessw-blog explores the critical distinction between a problem being theoretically solvable and it being successfully solved in practice. The discussion is anchored in a debate within the AI safety community regarding the \"hardness\" of aligning superintelligence. Specifically, it references a classification often discussed by researchers like Evan Hubinger and Chris Olah, where one potential scenario is that AI alignment is roughly as difficult as building the steam engine. In this view, alignment is a \"trivial\" or manageable engineering challenge that does not require discovering new physics or overcoming impossible barriers.
However, the author argues that this classification offers false comfort. The post posits that even if alignment is merely an engineering hurdle, humanity is currently on a trajectory to fail. To illustrate this, the author revisits the actual history of the steam engine-specifically the early, dangerous, and commercially failed attempts by Thomas Savery. Savery's initial pumps were prone to catastrophic explosions and were ultimately abandoned due to their inefficiency and danger. It took decades and subsequent inventors, such as Newcomen and Watt, to refine the technology into something safe and transformative.
The core argument presented is one of risk tolerance and the margin for error. The author contends that the current ecosystem of AI development treats the technology with a casualness that belies the stakes. If AI alignment is indeed a \"steam engine\" class problem, it still requires rigorous safety standards and immense, sustained effort to get right. The historical analogy highlights a terrifying asymmetry: when Savery's engine failed, he simply gave up, and the world waited for a better engineer. If the first deployed superintelligence fails due to \"solvable\" engineering oversights, there may be no opportunity for a second attempt.
This post serves as a sobering reminder for policymakers and technical leaders. It suggests that the debate over whether alignment is \"possible\" is secondary to the question of whether our current institutions are competent enough to execute the solution. The author concludes that without a radical shift in how seriously governments and labs treat extinction risks, even a technically solvable problem may prove fatal.
For those tracking the nuance of AI safety debates, this piece offers a vital perspective on why technical optimism should not be confused with safety assurance.
Read the full post on LessWrong
\n\n