In a recent post on LessWrong, the author explores the structural differences between calculating the risk of familiar engineering failures versus unprecedented existential threats.
\nIn a recent analytical piece on LessWrong, the author challenges the robustness of assigning specific probability estimates to unprecedented events, specifically focusing on potential catastrophes arising from advanced Artificial Intelligence.
In the field of AI safety, researchers and forecasters often attempt to quantify the risk of existential catastrophe, colloquially known as "P(doom)." These estimates range wildly, from near-zero to near-certainty. The discourse often treats these numbers as if they are comparable to engineering reliability metrics-such as the probability of a bridge collapsing or a rocket failing. This post argues that such a comparison is mathematically flawed because the underlying nature of the probability is different.
The core of the argument rests on a distinction between "canonical" and "non-canonical" probabilities. The author defines a canonical probability as a value that is stable across different, reasonable scientific frameworks given the available evidence. For example, despite different engineering philosophies, experts will generally converge on the failure rate of a specific steel beam because there is ample historical data and established physics to constrain the estimate.
Conversely, unprecedented catastrophes are "non-canonical." Because these events have never occurred, there is no historical frequency to force different priors to converge. The author utilizes Bayesian machinery-including likelihood ratios and algorithmic information theory-to show that estimates for these events are not measurements of the world, but rather reflections of the specific theoretical framework chosen by the estimator. In this context, a 10% risk estimate is not a stable feature of reality but a fragile artifact of a specific worldview.
This distinction is critical for decision-makers. If the probability of AI catastrophe is non-canonical, then debating whether the risk is 1% or 10% may be less useful than understanding the structural uncertainty of the model itself. It suggests that standard risk management approaches, which rely on stable probability distributions, may be ill-suited for mitigating existential risks.
For a deeper understanding of the Bayesian arguments supporting this distinction, we recommend reading the full analysis.
Read the full post on LessWrong
\n\n