Baidu has released ERNIE Evolve, an automated research system designed to improve AI training and inference code. The tool takes a repository, a specific objective, and evaluation criteria as input. It then selects research directions, edits code, and runs experiments. Only changes that pass a hidden evaluator and a separate reviewer are retained. The Baidu team tested the system on three public benchmarks. It subsequently optimised inference for ERNIE-5.5-Flash, which is still in development.
Quick facts
- Size: It is an agent system; Baidu has not disclosed which LLMs power its agents.
- Runs on: Benchmarks used 1 NVIDIA B200 (NanoChat, SOL-ExecBench) and 8×H100 (Speedrun Track 1). Production tests ran on Kunlunxin P800 and GPUs (GPU model not disclosed).
- Performance: Beat the best public result on 3 of 4 public metrics, and cleared production checks on ERNIE-5.5-Flash.
- Best: Generation speed on Kunlunxin P800 reached 623.6% of baseline after 4 research rounds.
- Worst: SOL-ExecBench mean SOL of 0.819, behind the 0.827 leaderboard best.
- Bottom line (best): Real production inference gains: +33.7% deterministic throughput, +14.9% decode speed.
- Bottom line (worst): Self-reported, system not released, and public-benchmark margins are small.
What is ERNIE Evolve?
ERNIE Evolve is an automated research system from Baidu that optimises code against a fixed, measurable objective. Baidu announced it on X as a practical step toward recursive self-improvement (RSI). In RSI, AI systems improve the software that builds and serves AI.
How does ERNIE Evolve work?
The system runs 2 nested loops. The outer loop decides what to investigate. The inner loop develops and tests improvements inside each direction.
- Director: picks the next research direction and a starting code version. It reads an Experience Bank of past results and a Knowledge Base of papers and code.
- Selector: chooses a candidate from a lane’s tree and names the next hypothesis to test.
- Executor: implements that plan in an isolated environment.
- Evaluator: a fixed program kept outside the repository. No agent can see it. Gains must exceed the noise threshold.
- Reviewer: checks whether a change follows the intended direction or exploits an evaluation loophole.
Several research lanes run in parallel, each with its own tree of code versions. Failed experiments are recorded, so later rounds can avoid repeating them.
How did ERNIE Evolve score on public benchmarks?
Baidu ran the same loop on NanoChat AutoResearch, NanoGPT Speedrun and SOL-ExecBench.
- NanoChat AutoResearch: 0.8763 validation BPB within a 5-minute, single-GPU budget. It added an n-gram branch and a document-retrieval branch beside the network, mixed by a learned gate.
- NanoGPT Speedrun Track 1: 38.9 s to target loss on 8×H100, versus the 39.9 s record. The results repo reports 38.92 s over 18 runs.
- NanoGPT Speedrun Track 3: 2,590 steps, versus 2,690. It decomposed weights into direction and magnitude, then added EMA-Nesterov extrapolation.
- SOL-ExecBench: 0.819 mean SOL across 235 kernel problems on B200, short of the 0.827 leader.
ERNIE Evolve vs other automated research systems
| Metrics | ERNIE Evolve (Baidu) | Recursive | Hyra-1.0 (Tencent Hunyuan) | (auto)²-research (rekursiv.ai) |
|---|---|---|---|---|
| NanoChat val BPB ↓ | 0.8763 | 0.9109 | 0.9015 | 0.8878 |
| Speedrun Track 1 ↓ | 38.9 s | 77.5 s | 76.4 s | Not reported |
| Speedrun Track 3 ↓ | 2,590 steps | Not reported | Not reported | Not reported |
| SOL-ExecBench, self-reported ↑ | 0.819 | 0.754 | 0.771 | Not reported |
| SOL-ExecBench, per Baidu (Eval Stack v1.1) ↑ | 0.819 | 0.676 | 0.736 | Not reported |
| Production use reported | ERNIE-5.5-Flash inference | Not disclosed | Science tasks (math, molecules) | Not disclosed |
| What is released | Final code + logs | Run artifacts | Results repo | Recipe + harness tools |
Recursive and Hyra-1.0 Speedrun times were set against the older ~79.7 s record, so they are not like-for-like. Hyra SOL self-report via Tencent coverage.
What did ERNIE Evolve do for ERNIE-5.5-Flash inference?
Baidu team then pointed the system at 3 live engineering tasks on ERNIE-5.5-Flash. Baselines had already been tuned by engineers.
- Low latency on Kunlunxin P800: generation speed rose to 623.6% of baseline over 4 rounds. Round 1 enabled fused mHC and speculative decoding via the model’s multi-token prediction layer (282.9%). Round 2 cut host-side overhead (588.8%). Round 4 fixed dispatch so DeepEP’s low-latency path engaged.
- Deterministic inference for RL: under SGLang’s deterministic mode, throughput rose 33.7% in about 3 days. Time per output token fell 26% and time to first token fell 12%. A Triton kernel for deterministic sampling, which used about 14% of GPU time, gave the largest single gain.
- Single-request decoding on GPUs: speed rose 14.9% with accuracy unchanged. 3 of 7 research directions produced gains. Key changes: FP16 instructions for tiny FP8 matmuls and a 2-level MoE expert scheduler.
What it means
For developers working on model infrastructure, this is a tool for internal optimisation. It does not change how customers use the API or what data is stored. The system operates on the engineering side to squeeze more performance out of existing hardware. Users do not see a direct interface to ERNIE Evolve; they only see the resulting speed improvements in the deployed models.
