Artificial Analysis has released the Search Index, a new benchmark that evaluates search API providers for AI agents based on quality, cost, and speed. The initial lineup tests Parallel, Exa, Firecrawl, You.com, Tavily, Keenable, and Brave using a standardised agent setup on the Stirrup framework. Each provider runs against GPT-5.6 Luna with 25 attempts per task to retrieve web pages. The index combines three equally weighted tests including DeepSearchQA with 900 research questions, a BrowseComp subset of 200 hard-to-find facts, and AA-Omniscience covering 600 questions across six knowledge domains. A tool-free baseline where the model answers without assistance provides the comparison point.
Better search quality lowers total costs because the model uses fewer tokens when it receives good results upfront. Parallel Search advanced reduces token use by over 40 percent compared to the Basic version, driving per-task costs down to $0.084 from $0.11 despite higher individual search charges. Raw speed per query does not always mean faster overall results. Parallel Search turbo clocks the shortest response time at 0.51 seconds against 1.03 seconds for Basic, but its lower quality score of 67 versus 73 forces the agent to run more passes. Total time per task winds up about the same. Artificial Analysis states Parallel, Firecrawl, and Parallel turbo hit the best mix of cost and performance. Other providers can apply to join the benchmark while the full methodology remains public.
- 25 runs per task ensure statistical reliability
- Three benchmarks weigh equally in the final score
- Open-source Stirrup framework standardises the agent setup
