METR Benchmark Signals 10x Annual AI Progress Rate
Coverage of lessw-blog
A new analysis from lessw-blog suggests that AI model capabilities, specifically regarding autonomous task duration, are now improving at a rate of 10x per year, significantly outpacing historical trends.
In a recent post, lessw-blog discusses a dramatic shift in the trajectory of Artificial Intelligence development as measured by the METR (Model Evaluation and Threat Research) Time Horizons benchmark. The analysis suggests that the rate at which AI models are improving their ability to handle long-duration tasks has accelerated significantly, moving from a historical average of approximately 3x per year to a staggering 10x per year.
To understand the gravity of this claim, it is important to contextualize the METR Time Horizons benchmark. Unlike static evaluations that test a model's knowledge on a multiple-choice exam (such as MMLU), METR focuses on agency and autonomy. It measures the "effective time horizon"-essentially, how long a model can work autonomously on a complex task before it requires human intervention or fails. This metric is considered a critical proxy for Artificial General Intelligence (AGI) because it evaluates the capacity for sustained, goal-directed behavior rather than momentary recall.
The post highlights that prior to 2024, the improvement rate on this specific metric was steady but manageable, doubling roughly every year (implying a ~3x annual improvement). However, with the release of recent reasoning-focused models and the METR Time Horizon 1.1 update, the data indicates a regime change. The analysis posits that time horizons are now doubling approximately every 3.5 months. This acceleration is attributed largely to advances in Reinforcement Learning (RL) scaling, which allows models to "think" longer and correct errors more effectively during inference.
The implications of a 10x annual improvement rate are profound. If this trend holds, the timeline to AGI-often defined in this context as the ability to automate high-value, long-horizon cognitive work-shortens considerably. The author notes that median timelines could now be as short as three years. This challenges many existing forecasts that rely on linear extrapolation of older scaling laws.
However, the analysis also offers a necessary caveat regarding the sustainability of this pace. The surge relies heavily on the efficiency of RL scaling. If these techniques encounter diminishing returns or if the "reasoning" capabilities prove brittle outside of specific benchmarks, the progress rate could revert to the previous 3x baseline by 2026 or 2027. Nevertheless, the current data signal represents a critical inflection point that researchers and policymakers must monitor closely.
For those tracking the velocity of AI development, this post provides a data-driven argument for why the next few years may be far more transformative than the last decade.
Read the full post at LessWrong
Key Takeaways
- AI progress on the METR Time Horizon benchmark has accelerated from ~3x to ~10x per year.
- Time horizons for autonomous model tasks are now doubling approximately every 3.5 months.
- This acceleration suggests median AGI timelines could be as short as three years.
- The surge is driven by recent model releases and updates to the benchmark methodology.
- A potential slowdown could occur in 2026-2027 if Reinforcement Learning scaling proves inefficient.