A recent statistical re-evaluation published on LessWrong suggests that current METR task data cannot definitively distinguish between exponential and superexponential growth, potentially skewing AI capability forecasts.
\nIn a detailed statistical analysis published on LessWrong, a contributor examines the dataset provided by METR (Model Evaluation and Threat Research) to assess the reliability of current AI forecasting models. As the AI community relies heavily on evaluation benchmarks to predict the arrival of advanced capabilities, the integrity and interpretability of this data are paramount. The post argues that the existing data points are insufficient to differentiate between fundamentally different growth trajectories, suggesting that our confidence in specific forecasting timelines may be misplaced.
The Context
METR (formerly associated with ARC Evals) provides crucial data used to track the progress of Large Language Models (LLMs) against human baselines. The standard assumption in many forecasting models is that AI progress follows a specific curve-often exponential. However, accurate forecasting requires not just a curve that fits past data, but one that accurately predicts future divergence. If multiple mathematical models fit the historical data equally well but predict vastly different futures, the utility of that data for long-term forecasting is compromised.
The Analysis
The author re-analyzed the METR task data using a Bayesian item response theory model. The central finding is that the data is currently too sparse or noisy to distinguish between exponential and superexponential growth. Specifically, the author demonstrates that four distinct trajectory shapes-linear, quadratic, power-law, and saturating-fit the existing data with similar accuracy. While they look the same looking backward, they diverge significantly when projecting forward, making precise timeline predictions difficult.
Furthermore, the analysis critiques the reported "horizon" numbers. The post claims that METR's calculation for the "80% success" horizon overstates current capabilities by approximately an order of magnitude. This discrepancy arises because the underlying model fails to account for the variation in task difficulty, assuming a uniformity that does not exist in practice. Consequently, the "effective horizon"-the point at which models reliably achieve high performance-is likely much further out than the raw metrics suggest.
Why This Matters
This critique highlights a significant blind spot in AI safety and strategy: the underestimation of uncertainty. The author points out that current credible intervals are too narrow because they treat human completion times as known constants rather than estimated variables. Without better data on human baselines and more robust statistical modeling, the community may be operating with a false sense of precision regarding how close we are to specific capability thresholds.
For researchers and forecasters, this post serves as a technical caution against over-fitting narratives to ambiguous data. It suggests that while the doubling time of capabilities (estimated here at roughly 4.1 months) is consistent with general expectations, the shape of the long-term curve remains an open question.
Read the full post on LessWrong
\n\n