Google Research has released Retrieve-for-Train (R4T), a system that speeds up complex AI search queries by 12 to 20 times. The method trains a small diffusion model offline to return multiple relevant results for a single broad request, rather than generating one single best match.
The problem with current fan-out methods
Search algorithms often split a broad prompt into several sub-queries to gather more information. This process is known as query fan-out. Google researchers found two main issues when standard large language models perform this task during live inference.
The first issue is paraphrastic collapse. When asked for ‘Bohemian festival style’, a zero-shot Qwen3-4B model generated ‘bohemian festival fashion’ and ‘festival bohemian clothes’. These near-synonyms lead to a homogeneous list of results that lack variety.
The second issue is latency. Generating text autoregressively and then calling the retrieval system repeatedly is slow. Best-of-N sampling improves quality but multiplies the inference cost.
The 3-step R4T pipeline
The new framework uses a specific three-stage process to solve these problems.
Fan-out LM training: A fan-out language model (FOLM) generates k sub-queries. A frozen dense retriever executes them. A set-level reward scores the whole retrieved set, not each item alone.
Supervision synthesis: The trained FOLM samples 128 fan-outs per query at temperature 0.9. These become (query, target set) training pairs with no human labels. For open-ended tasks, targets are retrieved-content embeddings. For compositional tasks, targets are sub-query embeddings.
Diffusive retriever training: A 53.9M-parameter diffusion transformer learns to map a query embedding to a full set of target embeddings. It uses a variance-exploding formulation within the EDM framework. At inference, it generates all embeddings in a single non-autoregressive pass. Nearest-neighbor search then maps each embedding to database items.
Reward design and reward hacking
For open-ended abstract retrieval (OAR), the reward combines 3 weighted terms:
- Groundedness (λg = 0.6): penalises distance between each sub-query embedding and its nearest database item.
- Diversity (λd = 0.2): the Vendi Score over representative retrieved items, such as the top-1 item per sub-query.
- Alignment (λa = 0.2): mean cosine similarity between each sub-query and the original query.
For weakly supervised compositional retrieval (WSCR), the reward is the fraction of reference-set items the fan-out retrieves.
The ablation study explains why all 3 OAR terms are important. With groundedness alone, Gemma3-4B converged to strings like ‘line ending line ending line ending.’ Adding alignment made collapse even faster, as the policy repeated paraphrases of the query. Adding diversity closed both shortcuts.
Training uses GRPO with soft PPO regularisation, which adds forward and reverse KL penalties. Key settings include group size 8, learning rate 1×10⁻⁷, and global batch size 512.
Results
Experiments used the Polyvore fashion outfit dataset with a CLIP-based matryoshka encoder at 128 dimensions. They also used a proprietary expert-playlist music dataset with MuLan embeddings. Every fan-out method produced k = 10 sub-queries, and Best-of-N used N = 5.
OAR quality was scored by an LLM judge on 5-point Likert scales. On Polyvore, Gemma3-4B R4T-FOLM averaged 49.1, versus 40.9 for Best-of-N and 38.5 zero-shot. Diversity rose from 56.0 zero-shot to 76.8. R4T-Diffusion retained most of it at 74.3. On Music, Gemma3-4B R4T-FOLM averaged 58.1 versus 49.2 for Best-of-N. Groundedness is not reported for R4T-Diffusion, since it produces no text sub-queries.
WSCR results on Polyvore show a coverage and diversity trade-off. R4T-FOLM (Qwen) reached 20.9 Recall@5K and 64.6 Hit@5K, versus 15.7 and 52.1 for Gemini-2.5-Flash. Its Vendi Score, however, dropped to 27.5. The authors link this to reduced output entropy under strong RL optimisation. R4T-Diffusion (Qwen) kept a higher Vendi Score of 34.7 with 16.5 Recall@5K.
Efficiency
At batch size 8, autoregressive fan-out took about 1.46 seconds. The diffusion model took 0.07 seconds. At batch size 1024, autoregressive fan-out reached nearly 50 seconds, versus 4.21 seconds for diffusion. The authors report a consistent 12× to 20× speedup.
What it means
For teams building search or recommendation engines, this changes the operational model. Instead of paying for repeated inference calls during peak traffic, the system trains a lightweight diffusion model once. This model generates all necessary search directions in a single pass, making the retrieval process significantly faster while maintaining diversity in the results.
