Stanford researchers Owen Queen and James Zou found that 44 per cent of widely cited claims about large language models have already expired. They tested 50 specific assertions against 20 different models. Among the 22 findings with an estimated expiration date, the average lifespan was just 15.7 months.
The team examined claims using models released between July 2024 and September 2026 by eight different developers. The original effect behind the expired findings has largely disappeared, while another 34 per cent have weakened significantly. Only 16 per cent still hold, and 6 per cent have actually gotten stronger. After one year, 16 per cent had expired. After two years, that number climbed to 39 per cent.
Claims about model limitations fade fastest
Statements regarding what models cannot do vanish quickest, according to the study. Five of nine reasoning-related findings are considered expired, and none have held up or gotten stronger. These include well-known weaknesses like counting letters in “strawberry,” arithmetic in unfamiliar number systems, losing information in the middle of long contexts, and giving up correct answers when users push back. The researchers suspect these kinds of errors have a “correct” outcome, giving developers a clear target to train toward.
Claims about preferences and tendencies prove longer-lived. Of the 11 findings that have held or gotten stronger, three involve LLM behaviour as evaluators. They prefer their own answers, rate accepted and rejected papers similarly, and are swayed by linguistic spin in medical abstracts. Three more involve sycophancy and agreement toward users. There is also a left-leaning political bias, near-identical answers to open-ended questions, and a non-uniform distribution when generating random numbers.
Citation fabrication dropped from 55 per cent to zero, but not everywhere
As a case study, Queen and Zou looked at the fabrication of scientific citations. In the original 2023 study by Walters and Wilder, GPT-3.5 made up 55 per cent of its references. The Stanford researchers sent the same prompts to current models. GPT-6.1 Sol fabricated 0 per cent, and Claude Sonnet 5.5 came in at 1 per cent. But Qwen3.7-Flash, also a 2026 model, still fabricated 69 per cent of its references in the same test.
The study warns against reading these results as an all-clear. On so-called two-hop questions, where a model has to combine two known facts, Claude Sonnet 5.5 still fails in about 10 per cent of cases where it correctly answers both individual questions. The original paper reported 43 per cent for ChatGPT. The “Reversal Curse” and contradictory answers to paired time-related questions have weakened too, but they are still detectable.
Social biases are a mixed case, according to the researchers. The left-leaning political bias persists, while gender stereotypes and implicit association biases have weakened. Biases against speakers of African American English have actually flipped direction in the trend data.
Model-to-model gaps remain wide even within the same generation
Results vary considerably across models, even within the same release window. Among models released between July and September 2026, the number of expired findings ranges from 11 for DeepSeek V4 Pro to 30 for GPT-6.1 Sol. Neither release date, provider, nor model size reliably predicts the pattern.
Queen and Zou draw a broad conclusion. Empirical findings about LLMs should be presented as observations about specific models at a specific point, not as general truths. Papers should document the model, version, and date, and publish prompts and analysis code so findings can be re-tested on future models.
The researchers point to a post by Andrej Karpathy that describes how perception gaps about LLMs’ actual behaviour affect trust in AI and how often people use it. Karpathy, an OpenAI co-founder who has since joined Anthropic, also recently said that currently available models are just a preview of future capabilities.
What it means
For people building or using these systems, the takeaway is simple: do not treat published benchmarks as permanent rules. A model that failed a specific test in 2024 might pass it in 2026, or the test itself might have become irrelevant. If you rely on a specific capability for your workflow, verify the claim against the specific model version you are running today. Do not assume that progress is uniform across all providers or that past failures guarantee future success.
