Rethinking Risk: Why Unprecedented Catastrophes Defy Standard Probability
Coverage of lessw-blog
In a recent post on LessWrong, the author explores the structural differences between calculating the risk of familiar engineering failures versus unprecedented existential threats.
In a recent analytical piece on LessWrong, the author challenges the robustness of assigning specific probability estimates to unprecedented events, specifically focusing on potential catastrophes arising from advanced Artificial Intelligence.
The Context: The Fragility of P(Doom)
In the field of AI safety, researchers and forecasters often attempt to quantify the risk of existential catastrophe, colloquially known as "P(doom)." These estimates range wildly, from near-zero to near-certainty. The discourse often treats these numbers as if they are comparable to engineering reliability metrics-such as the probability of a bridge collapsing or a rocket failing. This post argues that such a comparison is mathematically flawed because the underlying nature of the probability is different.
The Gist: Canonical vs. Non-Canonical Probabilities
The core of the argument rests on a distinction between "canonical" and "non-canonical" probabilities. The author defines a canonical probability as a value that is stable across different, reasonable scientific frameworks given the available evidence. For example, despite different engineering philosophies, experts will generally converge on the failure rate of a specific steel beam because there is ample historical data and established physics to constrain the estimate.
Conversely, unprecedented catastrophes are "non-canonical." Because these events have never occurred, there is no historical frequency to force different priors to converge. The author utilizes Bayesian machinery-including likelihood ratios and algorithmic information theory-to show that estimates for these events are not measurements of the world, but rather reflections of the specific theoretical framework chosen by the estimator. In this context, a 10% risk estimate is not a stable feature of reality but a fragile artifact of a specific worldview.
Why It Matters
This distinction is critical for decision-makers. If the probability of AI catastrophe is non-canonical, then debating whether the risk is 1% or 10% may be less useful than understanding the structural uncertainty of the model itself. It suggests that standard risk management approaches, which rely on stable probability distributions, may be ill-suited for mitigating existential risks.
For a deeper understanding of the Bayesian arguments supporting this distinction, we recommend reading the full analysis.
Read the full post on LessWrong
Key Takeaways
- Probabilities for unprecedented events (like AI doom) are structurally different from canonical probabilities (like bridge failures).
- A 'canonical probability' is defined as one that remains stable across reasonable scientific frameworks given available evidence.
- Unprecedented catastrophes yield 'non-canonical' probabilities because they lack the historical data necessary to force posterior convergence.
- The argument uses Bayesian machinery to demonstrate that estimates for novel risks are highly sensitive to the observer's chosen framework.
- This suggests that single-point estimates for existential risk are less reliable than engineering risk assessments.