The AI research organisation METR warns that current public benchmarks are insufficient to detect a potential rapid acceleration in model capabilities over the next few years. This gap in measurement could prevent humans from meaningfully assessing catastrophic risks posed by new systems. METR found that while models already outperform experienced professionals on specific coding and computational tasks, they have not yet surpassed large groups of top experts. For longer tasks requiring human time, the duration models can handle doubles every 3.6 months. A naive extrapolation of this data suggests they would achieve a 50 per cent success rate on the longest, 200-hour tasks by July 2027. The systems still struggle with open-ended questions lacking immediate feedback and situations difficult to recreate for training. METR also notes that common tests may overestimate real-world performance because developers train models in very similar test environments.
The core issue is that evaluation methods cannot keep pace with the speed of model improvement. If testing fails to identify dangerous capabilities early, safety interventions may arrive too late to prevent harm. This limitation applies specifically to the measurement of new systems rather than the behaviour of deployed products or customer accounts. The warning highlights a structural problem in how the field tracks progress on complex reasoning and long-horizon planning.
- Models double task duration capacity every 3.6 months
- Current benchmarks may miss dangerous acceleration
- Test environments often mimic training data too closely



