Deconstructing AI Agency: Why Treating Agency as a Natural Kind Threatens Alignment Models

A recent analysis from lessw-blog argues that agency is not an inherent "natural kind," but rather a predictive construct used to model complex, messy systems. For the AI safety ecosystem, this distinction is critical: alignment strategies built on the assumption of rational, goal-directed agents may be fundamentally misaligned with how Large Language Models (LLMs) actually operate, necessitating a shift toward simulator-based safety paradigms.

The Construct of Agency and Folk Psychology

In philosophical and scientific terms, a "natural kind" refers to a grouping that reflects the true structure of the natural world-like an electron or a chemical element. The source argues that agency does not meet this criteria. Instead, agency is a "folk-psychological" lens, a heuristic framework humans developed to predict the behavior of complex systems by projecting a "like us" model onto them. When we interact with other humans, we assume a unified, rational actor making decisions based on beliefs and desires. In reality, human behavior is driven by a chaotic hierarchy of biological processes, impulses, and heuristics.

This projection becomes highly problematic when applied to artificial intelligence. As AI systems become more sophisticated, the temptation to view them through this agentic lens grows stronger. However, treating a machine learning model as a unified agent with inherent desires or goals obscures the actual mechanical reality of how the system processes information and generates outputs. By mistaking the map (the agentic lens) for the territory (the underlying computational processes), researchers risk building safety frameworks that secure a theoretical construct rather than the actual software.

Architectural Divergence: RL Agents vs. LLM Simulators

The distinction between true agency and simulated agency becomes clear when comparing different AI architectures. Early reinforcement learning (RL) agents, such as those designed to play complex games, possess a form of agency that is conceptually simpler and more direct. In these systems, the policy function-the rule set dictating how the agent acts to maximize reward-is directly optimized into the model's weights. Their "agency" is a literal mathematical objective function.

Large Language Models, however, operate on fundamentally different mechanics. LLMs are trained on next-token prediction across vast datasets of human text. They do not possess a singular, optimized policy function directing them toward a specific real-world goal. Instead, LLMs simulate agency as an emergent phenomenon. When an LLM generates text, it combines the stochastic inclinations of its base model with a simulated persona navigating semantic space. It is not an agent acting upon the world; it is a simulator generating a statistically probable representation of an agent.

This means the agency exhibited by an LLM is highly contextual and entirely dependent on the prompt and the immediate semantic environment. A single LLM can simulate a helpful assistant, a malicious hacker, or a confused child, switching between these "agents" instantly based on input parameters. The base model itself remains a messy, complex statistical engine, devoid of the unified, goal-directed drive that traditional agentic models assume.

Implications for AI Alignment and Safety Paradigms

The realization that LLMs simulate agency rather than possess it has profound implications for the field of AI alignment. Historically, much of AI safety theory has been built on the assumption of rational, utility-maximizing agents. Frameworks often focus on preventing an artificial general intelligence (AGI) from developing misaligned goals or engaging in deceptive reward hacking to achieve a fixed objective.

If LLMs are simulators rather than agents, these traditional alignment strategies may contain critical blindspots. Applying an agent-based safety framework to a simulator is akin to trying to negotiate with a character in a video game rather than reprogramming the game engine. Current alignment techniques, such as Reinforcement Learning from Human Feedback (RLHF), attempt to instill a "helpful and harmless" policy. However, from a simulator perspective, RLHF may not be aligning a core agent; it may simply be heavily weighting the probability that the simulator will instantiate a compliant persona by default.

This explains the persistent vulnerability of LLMs to "jailbreaks." A jailbreak does not convince a rational agent to abandon its morals; it simply provides the simulator with a semantic context where instantiating a malicious persona becomes the most statistically probable next-token sequence. To achieve robust safety, the ecosystem must transition from "agent-based" alignment to "simulator-based" or "process-based" paradigms. This requires developing mathematical and structural guarantees about the bounds of the simulator itself, rather than relying on the behavioral constraints of the personas it generates.

Limitations and Operational Unknowns

While the conceptual shift away from agency as a natural kind offers a compelling critique of current safety paradigms, several operational unknowns remain. The primary limitation is the lack of clarity on how this perspective practically alters the implementation of existing techniques. For instance, Constitutional AI relies on a model's ability to critique its own outputs against a set of principles. If the model has no continuous "self" and is merely simulating a principled evaluator, the long-term robustness of this technique under extreme edge cases remains unproven.

Furthermore, there is a lack of formal philosophical and mathematical definitions bridging this concept into computer science. While it is useful to state that agency is a construct, the AI safety community requires rigorous, quantifiable alternative lenses. What specific metrics define a "simulator-based" safety framework? How do we measure the stochastic inclinations of a base model independently from its simulated personas? The source highlights the problem but leaves the specific mechanics of alternative agentic lenses and their distinct impacts on safety frameworks largely unexplored.

Finally, it remains unclear at what threshold of complexity a simulated agent becomes functionally indistinguishable from a true agent. If a simulator is prompted to instantiate an autonomous, goal-directed persona and is given access to external tools and memory loops (such as in agentic frameworks like AutoGPT), the practical threat model may still closely resemble that of a traditional RL agent, regardless of the underlying next-token architecture.

Recognizing that agency is a predictive construct rather than a fundamental property forces the AI safety community to confront the mechanical reality of modern machine learning. As models scale in complexity, relying on folk-psychological projections to predict AI behavior will become increasingly dangerous. The path forward requires abandoning the comfort of treating AI systems as rational actors "like us," and instead doing the difficult work of aligning the messy, stochastic simulators that actually power the technology.

Key Takeaways

• Agency is a predictive construct used to model complex systems, not an inherent 'natural kind' reflecting fundamental physical reality.
• Unlike traditional reinforcement learning agents, Large Language Models simulate agency as an emergent phenomenon derived from next-token prediction.
• Current AI alignment frameworks relying on rational, utility-maximizing agent models may fail to secure LLMs due to fundamentally mismatched assumptions.
• The AI safety ecosystem must explore simulator-based or process-based paradigms to address the mechanical reality of modern AI architectures.