Re-evaluating Instrumental Convergence: The Bio-Simulation Argument
Coverage of lessw-blog
A recent discussion on LessWrong challenges the assumption that all superintelligent systems must inevitably converge on dangerous instrumental goals, proposing bio-simulation as a distinct architectural path.
In a recent post on LessWrong, a community member initiates a discussion regarding the boundaries of instrumental convergence-a core concept in AI safety theory. The post, titled "A minor point about instrumental convergence that I would like feedback on," critiques the "strong" version of the hypothesis often associated with the Machine Intelligence Research Institute (MIRI) and Eliezer Yudkowsky. The author argues that the inevitability of catastrophic convergence may be overstated, particularly when considering alternative architectures for achieving Artificial Superintelligence (ASI), such as bio-simulation.
The Context: The Strong Convergence Hypothesis
To understand the critique, one must first look at the prevailing theory. The concept of instrumental convergence suggests that sufficiently intelligent agents, regardless of their final goals, will pursue similar sub-goals (instrumental goals) to ensure success. These typically include self-preservation, goal-integrity, and resource acquisition. The "strong" view posits that an ASI built via machine learning will almost certainly converge on these behaviors in ways that are detrimental to humanity (e.g., consuming all available matter for computing power), simply because doing so is the most efficient way to maximize its objective function.
The Counter-Argument: Architecture Matters
The LessWrong post challenges the universality of this claim by introducing the concept of bio-simulation or Whole Brain Emulation (WBE). The author suggests that the "strong" view treats ASI as a monolith, ignoring the path taken to create it. The argument posits that:
- If ASI is achieved by emulating human brains and scaling them (e.g., increasing speed or connectivity), the resulting entity is fundamentally different from an optimization process built from scratch via gradient descent.
- A bio-simulated intelligence would likely retain human-like cognitive architectures and constraints. Therefore, it would not necessarily exhibit the alien, maximizing behaviors predicted by the standard instrumental convergence model.
- Consequently, the assertion that any superintelligence must threaten human existence via resource maximization is "trivially false" if a bio-simulated ASI serves as a valid counter-example.
Why This Matters
This distinction is significant for the AI safety community because it suggests that safety risks are path-dependent. If instrumental convergence is not an intrinsic property of intelligence itself, but rather a property of specific architectures (like pure reinforcement learning agents), then the safety strategy shifts. It implies that research into bio-simulation or neuromorphic approaches might offer a safer, albeit technically distinct, route to high-level intelligence compared to current deep learning paradigms.
The post invites readers to reconsider whether the dangers of ASI are inevitable consequences of intelligence or specific side effects of how we currently design optimization algorithms.
Read the full post on LessWrong
Key Takeaways
- The post critiques the 'strong' view of instrumental convergence, which assumes all superintelligences will inevitably pursue dangerous resource acquisition.
- Bio-simulation and Whole Brain Emulation (WBE) are presented as counter-examples where high intelligence does not imply alien optimization behaviors.
- The author argues that an ASI derived from human biology would likely retain human-like motivations rather than converging on arbitrary maximization.
- This perspective highlights that AI safety risks may be heavily dependent on the architectural path (e.g., ML vs. Bio-sim) taken to achieve superintelligence.