Keenable AI has released NEEDLE, a live benchmark that rebuilds its search queries every hour to stop models from memorising answers. The tool generates news queries from RSS feeds and Google Trends, while finance, legal, and academic queries are refreshed daily from sources like SEC XBRL and CourtListener. Fifteen search APIs are tested against the same text, with scores judged by ultimate, a pooled oracle engine that represents the best possible retrieval the entire field has managed to achieve.
Is it reproducible
Yes. The project is an open-source evaluation harness, not a commercial product. Developers install the Python CLI using uv sync and run two subcommands: generate and run. The system requires an OpenRouter key for judging and one API key for each engine being tested. It runs on a standard laptop or within a CI environment. All query streams and ranking quality judgements can be recreated.
What NEEDLE measures
The benchmark covers five verticals: News, Everyday, Expert, Deep-tail, and Legal. Each category models a specific agent intent. News projects the latest items from roughly 124 curated RSS feeds and Google Trends into keyword queries. Finance requests registry facts from Wikidata and GLEIF, alongside single-quarter 10-Q figures from SEC XBRL. Scholar converts one paper into four query styles: a degraded title, a full-text-only detail, a natural-language clue, and a hedged tip-of-the-tongue description. Deep-tail samples rare-word queries from public agent-trajectory releases including DeepResearchGym, OpenResearcher and LRAT. Legal pulls recent CourtListener opinions across 14 federal courts and eCFR sections.
Scoring methods vary by category. News and deep-tail have no single correct result, so an LLM judge rates each result from 0 to 4, with the harness reporting nDCG@5 and applying a duplicate-URL penalty. Finance reports answer-recall@5, checking if the fact reaches the agent inside a top-5 snippet. Scholar and legal are known-item tasks scored by identifier match.
The ceiling is the interesting part
Every engine receives identical query text. The runner issues one call at a time to ensure latency percentiles are comparable and no engine takes concurrent load. Judging occurs on the engine’s own ranking, titles and snippets. Pages are never fetched and results are never re-ranked. Evidence is clipped to 2,000 characters for everyone, and the judge does not see the engine name.
The more important number is the ultimate ceiling. For each query, NEEDLE pools the results returned by every engine into a synthetic oracle engine, then orders that combined set by relevance. This creates an empirical ceiling based on what the entire field was able to retrieve.
The gap to ultimate acts as an upper bound on agentic search quality as it stands today. A large gap means better results existed but every engine failed to surface or rank them well. A weak ultimate score indicates something different: even after pooling every provider, the benchmark found little strong evidence. In short, NEEDLE can distinguish a ranking problem from a retrieval problem shared by the whole market.
Where the field actually stands
Numbers below are published 7-day means for the window ending 2026-08-28.
Finance is close to solved: Exa scores 0.910, Keenable 0.872, Perplexity 0.871, and Google 0.847, against an ultimate of 0.965. Scholar spreads out, with Keenable at 0.774 and Tavily at 0.310 against a 0.869 ceiling, because title queries are answerable from metadata while body queries are not. Deep-tail is hardest and closest to real agent traffic: Exa leads at 0.557 of ultimate, Keenable follows at 0.470, and Bing sits at 0.199. The gap between delivered and achievable quality widens as queries approach how agents actually search.
Latency is another important metric here, because agents call search dozens of times per task. Same window: Keenable-realtime is 193 ms p50 / 284 ms p95, Exa is 1,876 / 2,955, and Bing is 2,767 / 9,381.
What it means
For developers building search agents, this benchmark removes the ability to cheat by memorising static datasets. It forces tools to demonstrate actual retrieval capability against fresh data. The ultimate score provides a clear target for improvement, showing whether a tool is failing to find information or simply failing to rank it correctly.
