In a recent analysis, lessw-blog explores the foundational mechanics of decision theory, arguing for the necessity of Updateless Decision Theory (UDT) in creating consistent, rational agents.
\nIn a recent post, lessw-blog discusses the intricate landscape of decision theory, specifically focusing on the tension between Updateless Decision Theory (UDT) and traditional "updateful" models like Causal Decision Theory (CDT) and Evidential Decision Theory (EDT). The author aims to refine arguments supporting UDT, suggesting that resistance to the theory often stems from overlooking specific logical necessities regarding how agents process information and commitment.
The Context: Why Decision Theory Matters for AI
While decision theory often appears abstract, it is the operating system of rational agency. Standard theories (CDT and EDT) dictate how an agent chooses actions based on causal links or evidence. However, these standard models can fail in scenarios involving self-reference or prediction-classic examples include Newcomb's Problem or Parfit's Hitchhiker. In the context of Artificial Intelligence, these failures are not merely philosophical puzzles; they represent potential vulnerabilities. An AI that follows naive updateful logic might be susceptible to manipulation or fail to maintain goal stability when building successor agents (the "tiling problem").
The Gist: Logical Consistency as a Driver
The post serves as a preliminary exploration for a "next tiling theorem," indicating the author is working on formalizing how agents can reliably construct successors that share their values. The core argument presented is that decisions should be viewed through the lens of logical consistency. The author draws inspiration from the concept that "decisions are for making bad outcomes inconsistent."
This perspective challenges the intuition that an agent should always update its beliefs and actions based on the most recent information. Instead, UDT suggests that to achieve the best outcomes, an agent must sometimes act according to a policy determined prior to receiving specific inputs-effectively ignoring new data to honor a pre-commitment. The author argues that naive updateful theories fail to account for this, leading to suboptimal or incoherent behaviors in complex environments.
Why Read This?
For researchers and engineers involved in AI alignment, game theory, or agent architecture, this post offers a critical look at the mechanisms of rational choice. It moves beyond the basic definitions of UDT to explore the underlying intuitions that make "updatelessness" a requirement for robust agency.
Read the full post at LessWrong
\n\n