Google Research has released ToolGrad, a new framework that achieves a 99.8% pass rate for generating tool-use data. The method inverts the standard pipeline by constructing verified tool chains first and then writing the corresponding user queries.
Researchers from Google, the University of Tokyo, RIKEN AIP, and Tohoku University have made the code available under Apache-2.0. The ToolGrad-500 dataset is hosted on Hugging Face, alongside models at 1B, 4B, and 12B parameters. A PyPI package is also available for installation.
The problem with query-first generation
Previous pipelines like ToolBench and ToolACE follow a query-first recipe. A system samples a pool of APIs, asks an LLM to invent a plausible user instruction, and then dispatches a depth-first search agent to find a tool-use path that satisfies it.
The search has no guarantee of success. When it dead-ends, the compute spent on exploration is wasted, and the sample is discarded. The paper frames this as distilling valuable trajectories from a complex and often failing agent exploration, which is inherently inefficient.
ToolGrad reverses the order. It first constructs a ground-truth tool-use chain by actually executing APIs, then annotates that chain with a matching user query. An explicit, working chain is far less ambiguous than a hypothetical prompt, so the chain-to-query step takes a single LLM call.
Four modules in a loop
Each iteration runs four modules in sequence:
- API Proposer narrows a sampled set of APIs down to a few candidates that could extend the current workflow.
- API Executors run those candidates in parallel and produce detailed execution reports.
- API Selector reviews the reports, picks the single best-performing call, and appends it to the workflow. Its directional feedback is the textual gradient.
- LLM Updater rewrites the synthetic user query and AI response so they match the new API set.
Repeating the loop yields one sample: a user query, a verified API workflow, and the final response. The repository’s default configuration runs 10 iterations over 50 sampled APIs per workflow.
Generation efficiency on ToolBench
The research team evaluated data generation on the ToolBench API database, which contains 16,000+ real-world APIs, and compared ToolGrad against ToolBench’s DFS-based query-first approach. According to the research paper:
- Pass rate rose from 63.8% (DFS) to 99.8% (ToolGrad).
- Ground-truth tool uses per sample rose from 2.1 to 3.4, meaning longer chains.
- Tool-use steps per sample fell from 34.3 to 20.0.
- LLM invocations per sample fell slightly, from 64.5 to 63.9.
The 0.2% failure case occurred when the agent could not get a successful response from 3 selected APIs across all 10 iterations and saved an empty sample.
BFCL results with Gemma-3
The researchers generated ToolGrad-500, a 500-sample dataset built with Gemini 2.5 Flash-Lite, and used it to post-train Gemma-3 at 1B, 4B, and 12B parameters. They evaluated on the Berkeley Function Calling Leaderboard, which uses a tool set that differs from ToolBench, making it an out-of-distribution test with unseen tools.
Findings reported by the authors:
- Fine-tuning on ToolGrad-500 improved tool-use scores at every parameter size.
- ToolGrad-12B scored 83.1, compared with Gemini 2.5 Pro at 83.2, Claude 4.5 Opus at 82.8, and GPT-5 at 74.4, as measured at the time of publication.
- The 12B student outperformed Gemini 2.5 Flash-Lite, the teacher model that generated its training data.
- ToolGrad-12B led open tool-use specialists including ToolACE and Hammer-2.1-7B.
The repository’s reproduction scripts target BFCL V1 and V2 through a customized fork, run inference in a vLLM Docker image, and were verified on a single NVIDIA A100 40GB.
What it means
The practical impact is a drastic reduction in wasted compute. Developers no longer need to rely on blind search agents to find valid API sequences, a process that previously discarded the majority of generated samples. With a 99.8% pass rate, training data becomes significantly cheaper and faster to produce. Furthermore, a small dataset of just 500 samples allows a 12B parameter model to reach performance levels comparable to much larger proprietary systems. This suggests that high-quality synthetic data generated through verified execution is more effective than raw scale.
