A recent METR study finds AI has accelerated software vulnerability discovery but has not yet driven measurable progress in optimising AI research itself.
Where LLMs are actually helping science
The analysis breaks down contributions across three fields: cyber, mathematics, and AI research. The results show uneven acceleration.
Cyber vulnerabilities: Major acceleration
The rate of reported vulnerabilities has dramatically increased in 2026 compared with 2025. This holds true for specific projects like cURL, OpenSSL, Firefox, and Microsoft, as well as aggregate databases such as the US NVD and OSV.
Mathematics research: Minor acceleration
AI is clearly contributing to higher output volumes, with arXiv submissions doubling in certain areas in under 12 months. However, quantifying the actual value remains difficult. Several problems from prestigious lists have been solved, including the Jacobian conjecture from Smale’s list, Problem 44 from Green’s list (the halving sieve), and the sofic half of Green’s Problem 100. It is too early to determine if this trend will sustain.
Optimization of AI research: No measurable acceleration
When examining algorithmic progress across seven significant problem areas—CIFAR-10, Hutter compression, Gurobi mixed-integer programming, MIPLIB, nanoGPT, Stockfish, and the matrix-multiplication exponent—LLM-attributable contributions are rare. Usage rates here are far lower than in cybersecurity or mathematics.
The paper suggests acceleration occurs when models undergo a phase change for a specific skill, evident in coding and cyber in recent years. The key question is whether similar shifts will occur in other scientific domains.
Read more: Research note: Have We Seen an Acceleration in Discoveries? (METR)
Automating environment generation with SPADE
A multi-university group has developed SPADE, a framework for generating synthetic data in the form of game-like environments. LLMs can train within these environments to bootstrap the creation of further data.
The team
Researchers from the University of Washington, Stanford University, Northeastern University, Carnegie Mellon University, the Massachusetts Institute of Technology, the National University of Singapore, Seoul National University, Stevens Institute of Technology, and the University of Chicago built the system.
How it works
SPADE allows an LLM to alternate between creating executable training environments and solving them. The model performs two roles:
- Environment Designer: Writes complete, long-horizon training environments as executable code.
- Reasoning Agent: Learns to act in the environments. The reward is estimated by measuring the gap between performance with and without privileged hints. A hint is task-relevant information, such as a partial solution sketch, attached by the designer.
Performance at scale
The authors trained three Qwen3 backbones: Qwen3-4B-Instruct-2507, Qwen3-8B, and Qwen3-30B-A3B-Instruct-2507. The 30B model performed best. Each was tuned via GRPO for 400 rollouts of 25 environments. Assessments against benchmarks like AIME, GPQA, LCB, and Reasoning Gym showed improvements in both game and tool-use environments.
At the 30B-A3B scale, SPADE reached a suite average of 58.3, an increase of 8.1 over the base model and 5.3 over the strongest fixed-environment baseline. Applying the same method to tool-use design improved every backbone tested.
This approach functions as a form of synthetic data generation, allowing researchers to use available powerful models to create diverse training sets. It reduces the cost of building broad datasets. However, the authors note it cannot bootstrap models beyond the imaginative capabilities of the base model used for environment generation.
“By representing environments as Python programs with a Gym-style interface, the framework unifies single-turn reasoning and multi-turn agentic tasks, and turns environment design into a learnable, RL-trained component of post-training, enabling continual open-ended self-improvement,” the authors write.
Read more: SPADE: Self-Play in Adaptive Synthetic Executable Environments (arXiv)
Get the code here, including model checkpoints: SPADE (spade-rl, GitHub)
Building better GPU kernels with Hawkeye
Researchers from Harvard, Stanford, Together AI, and Caltech have built Hawkeye to help agents write well-optimized kernels for specific GPU hardware. The goal is to make coding agents hardware-aware with minimal expert intervention.
The core contribution
Hawkeye is an open-source framework that grounds autonomous kernel generation in a minimal taxonomy. It demonstrates that minimally supervised coding agents can exploit architecture-specific hardware features.
The key innovation is a generalizable set of unit tests. Each test pairs a human-authored solution kernel with a profiling metric that verifies the optimization. The solution kernel is wrapped as a callable function with a short usage guide, allowing agents to read it as a syntax example or invoke it directly.
Results
The team evaluated Hawkeye on porting PyTorch workloads across NVIDIA Ampere, Hopper, Blackwell, and AMD MI350 architectures, using BF16, FP8, NVFP4, and MXFP4 precisions.
On established workloads where torch.compile dispatches to expert-tuned vendor libraries like cuBLAS, cuDNN, and FlashAttention, Hawkeye matches or exceeds performance in both BF16 and low precision, including formats PyTorch cannot natively run.
On emerging attention variants where torch.compile cannot fuse non-standard scans and gates, Hawkeye reached an 18.9× geomean speedup against expert-authored Triton kernels from the Flash Linear Attention library. It approached or exceeded FLA on Linear Attention across every architecture, including 1.22× on Blackwell and 1.00× on MI350.
Scaling test-time compute with Hawkeye consistently generated the most performant kernels across architectures.
What it means
Papers like this show that with a small amount of human-curated knowledge, AI systems can learn to match and exceed highly-optimized human work. The lesson is that we can create gold-label helper systems, and machines will use them to bootstrap capabilities beyond our own.
Read more: Hawkeye: Hardware-Aware GPU Kernel Optimization with Minimal Supervision (alphaxiv)
