AllenAI has released Old Ideas, Novel Problems, a new benchmark designed to test how reliably large language models can judge whether a scientific research idea is new. The team shares the code, data, and an automated pipeline for building custom evaluation sets alongside the paper.
At Ai2, researchers have built systems like DataVoyager and AutoDiscovery to help scientists explore data and generate hypotheses. These tools aim to move researchers from understanding existing work to proposing and testing new theories. But generating an idea is only half the task. The system must also know if that idea contributes something genuinely new or simply repackages old work.
Usually, another LLM acts as a novelty judge. These models decide if an idea is new relative to prior work, either alone or by comparing it to another idea. There is no standard way to build one; teams pick a model, a prompt, and an evaluation format. Those choices change the verdict. Often, these judges are tested only on human-written papers, not the AI-generated ideas they are actually meant to evaluate.
The new benchmark results suggest novelty evaluation is far from reliable. Across six LLM judges – GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6 – small changes to the setup produce dramatically different judgments. In one experiment, changing a few words in a prompt shifted a model’s accuracy on identifying the more novel of two ideas by more than 50 percentage points. Retrieval and extra reasoning did not consistently improve results. Two purpose-built novelty evaluation systems performed worse than a simple prompted Claude Opus 4.6 baseline.
If AI is to play a larger role in scientific discovery, we need dependable ways to tell if it is actually proposing something new. This work provides tools for studying that problem while showing how much remains unresolved.
Building a benchmark for novelty judges
To test how reliably novelty judges can distinguish more novel research ideas from less novel ones, the team first needed examples where there was strong evidence about the originality of the underlying scientific contribution.
The pipeline is built around ICLR 2026 peer reviews. The team mined the reviews themselves for passages where reviewers explicitly discussed a contribution’s originality. A paper can be accepted without being particularly novel as long as it is strong enough in other aspects. Likewise, a rejected paper with an original contribution can have other weaknesses.
The team created a high-novelty pool of 154 human-authored research ideas. These were accepted papers in the top-rated 10% of their ICLR research area whose reviewers positively identified novelty and where no reviewer disputed it. The reverse criteria built a lower-novelty pool: 145 ideas from rejected papers in the bottom 10% whose reviewers raised originality concerns without any reviewer positively identifying novelty. An annotator agreed with the extracted novelty-signal polarity in 47 of 50 cases during a separate check. None of the disagreements would have changed whether a reviewer was counted as praising or disputing the paper’s novelty.
Together, these two pools form the first evaluation setting, Human-Only. This tests whether novelty judges can tell apart human-authored ideas with strong evidence of higher or lower novelty. In practice, novelty judges are increasingly being used to evaluate AI-generated ideas. The team created a second setting, Human+Generated.
For this pool, the team paired the same 154 validated human ideas with 154 ideas produced by a simple LLM generator. The generated ideas were assumed to be less novel than their human-authored counterparts. These are weak labels, not verified judgments for every idea. The generator received only a research area and an instruction to propose a novel idea, with no literature search, tools, iteration, or feedback. This gave a conservative comparison between ideas from a basic generator and contributions from top-10% ICLR papers whose reviewers had explicitly endorsed their originality.
As additional checks, an expert independently reviewed 30 Human+Generated pairs. The expert heavily weighted cases where the strongest judge got wrong. The team rewrote the human- and AI-generated ideas into a common structure to test whether judges were simply reacting to differences in writing style. The expert preferred the human idea in all 30 pairs and judged none of the generated ideas novel.
Importantly, many core findings do not depend on the labels. If two reasonable evaluation setups give opposite answers for the same pair of ideas, that disagreement itself reveals instability.
Small changes can produce very different verdicts
Using the collections of human-authored and AI-generated research ideas, the team tested how well six LLM judges – GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6 – could evaluate novelty through both pointwise and pairwise comparisons. The team varied one factor at a time: the prompt, reasoning budget, whether the judge could see related papers, how the ideas were formatted, how many judgments were combined into a final verdict, and for AI-generated ideas, which model generated them.
The largest shifts in performance came from how the team prompted the model to judge novelty.
For example, the team showed GPT-5.4 the same Human+Generated pairs. Each pair contained one validated human-authored idea and one AI-generated idea. One prompt told the model that reviewers had found exactly one idea in each pair to be novel and asked it to identify which one. The other simply asked which idea was novel. When the team switched to the simpler prompt, accuracy dropped from 93.5% to 40.9% – below chance – and the model changed its verdict on more than half of the same pairs.
By contrast, the same prompt change shifted accuracy by only about seven percentage points on the Human-Only dataset. That suggests a novelty judge can appear relatively stable on human-authored research but behave very differently on AI-generated ideas.
Human-authored ideas were not immune. The team saw substantial shifts on the Human-Only dataset as well. The same change moved Claude Opus 4.6 accuracy by 21 points. The shifts were just less severe than on the AI-generated ideas.
There was no universally better prompt. For example, when judging ideas individually, dropping an instruction that warned judges not to treat trivial or incremental contributions as novel improved accuracy for some judges while reducing it for others. A prompt that works well for one model may not transfer to another.
The team also tested a common use of novelty judges: comparing ideas from two AI generators. The team paired ideas from Claude Opus 4.5 with Claude Sonnet 4.5, and GPT-5.4 with GPT-5.1. Across the tested prompt changes, roughly a quarter to a third of verdicts changed on average across six judges. This disagreement can be measured without assuming which generator produces more novel ideas, showing how the choice of judge prompt can affect comparisons between ideation systems.
More search and reasoning did not necessarily produce better novelty judgments. The team gave GPT-5.6 Sol an extra-high reasoning budget and live, iterative search restricted to arXiv. The team then compared it with standard-retrieval judges that each saw five pre-retrieved abstracts. All judges in this comparison ran once per idea on Human+Generated. The agentic judge scored 0.78 macro-F1 – 11 points behind Claude Opus 4.6 at 0.89, and in roughly the same range as simpler Claude Opus 4.5 and GPT-5.4 setups – while costing roughly 10× more per decision at current API prices. Macro-F1 measures how well a judge identifies novel and non-novel ideas; higher is better.
Purpose-built evaluators were not more accurate either, despite costing substantially more. On the Human+Generated set, a simple Claude Opus 4.6 prompt using low reasoning and majority voting across three calls achieved 0.854 macro-F1 at a total evaluation cost of $2.59. By comparison, Idea Novelty Checker, a literature-grounded retrieval and reranking pipeline, scored 0.642 at $30.66. Sakana AI’s AI Scientist, whose novelty evaluator iteratively searches Semantic Scholar, scored 0.772 at $82.05. Both dedicated evaluators used Claude Opus 4.6, with one call per decision.
In other words, the simple prompted baseline was both more accurate and considerably cheaper than either purpose-built system.
Taken together, these results suggest that novelty evaluation is particularly brittle on the AI-generated ideas these systems are increasingly being asked to judge.
Toward more reliable evaluation of novel research
The results suggest that LLM-based novelty evaluation remains a major open problem. Judges that appear reliable on human-authored research may behave differently on AI-generated ideas. Prompts that help one model can hurt another. More search or compute does not necessarily produce more consistent judgments.
Future work should test judges on the kinds of ideas they will actually evaluate, measure sensitivity across prompts and models, improve how prior literature is incorporated, and build benchmarks that can evolve as the scientific literature grows and AI systems improve. The benchmark and reusable data pipeline are a starting point. Researchers can apply them to new peer reviews, other scientific communities, and future generations of novelty judges.
Automated ideation depends on more than generating plausible research directions. The team also needs ways to tell when those ideas are genuinely new. By making the weaknesses of current novelty judges easier to measure, the release helps researchers build better ways to evaluate AI-generated research ideas.
