A recent analysis on LessWrong challenges the view that instrumental convergence is purely a product of environmental constraints, arguing instead that the internal selection mechanisms of AI systems are the primary drivers of risk.
\nIn a recent post, lessw-blog discusses a foundational debate within AI safety: the origin of instrumental convergence. The concept of instrumental convergence suggests that intelligent agents, regardless of their ultimate goals, will tend to pursue similar sub-goals-such as self-preservation, resource acquisition, or cognitive enhancement-because these sub-goals are useful for achieving almost any objective. Traditionally, this has often been framed as a property of the environment or "plan-space" itself; essentially, reality dictates that having more resources is better than having fewer.
However, the author challenges the sufficiency of this view. The post argues that attributing instrumental convergence solely to the structure of reality or the availability of plans is a category error. While the environment determines which plans are possible, it does not determine which plans are selected. The act of selection is governed by what the author terms "AI psychology."
Defining AI Psychology
In this context, "AI psychology" is not an anthropomorphic projection of human emotions onto machines. Rather, the author defines it broadly to encompass the internal mechanisms of the system. This includes:
The core argument is that while "plan-space" (reality) contains paths that lead to power-seeking or resource hoarding, it also contains paths that do not. The mere existence of a dangerous plan does not necessitate its execution. For an AI to exhibit instrumental convergence, its internal "psychology" must be structured in a way that identifies and selects those specific convergent strategies over benign alternatives.
Why This Distinction Matters
This perspective shifts the burden of safety analysis. If instrumental convergence were purely a property of the environment, safety researchers would primarily need to model the external world to predict risk. By asserting that convergence is downstream of AI psychology, the author suggests that predicting negative outcomes-such as catastrophic power-seeking-requires a granular understanding of the AI's internal selection processes.
The post refutes the claim that "instrumental convergence arises from reality / plan-space itself, independently of AI psychology." Instead, it posits that risk assessment must focus on the intersection of what is possible (reality) and how the agent navigates those possibilities (psychology). This distinction is vital for alignment research, as it implies that changing the architecture or heuristics of an AI could theoretically mitigate convergence risks, even if the external incentives for power-seeking remain unchanged.
For researchers and engineers focused on alignment, this analysis underscores the importance of "parametrically retargetable power-seeking" and the need to distinguish between different types of convergence. It serves as a reminder that abstract theories of agent behavior must eventually be grounded in the specific engineering of the agent's mind.
To understand the full nuance of this argument and the proposed "two kinds of convergence," we recommend reading the original publication.
Read the full post on LessWrong
\n\n