Stanford audit finds LLM claims expire in 15.7 months on average

What expires fastest: claims about what models cannot do

Queen and Zou ran the 50 claims against 20 models from eight developers, released between July 2024 and September 2026. The results are far from uniform.

That matters to anyone who cites a benchmark. The evidence behind such claims lasts about a year and a quarter. Researchers who build on published findings are aiming at a moving target. So are buyers who pick models from published comparisons. No agreed way exists to mark a claim as expired. Readers must check the model and the date themselves.

The split across the 50 claims was uneven. In all, 44 percent had expired, 34 percent had weakened significantly, 16 percent still held and 6 percent had grown stronger. Among the 22 findings with an estimated expiration date, the average lifespan was 15.7 months. Decay also speeds up. After one year, 16 percent of findings had expired. After two years, the figure reached 39 percent.

Claims about what models cannot do go stale fastest.

Five of nine reasoning-related findings are expired. None held up or got stronger. The list reads like a tour of familiar weaknesses. Counting the letters in strawberry is one. Arithmetic in unfamiliar number systems is another. So is losing information in the middle of long contexts. So is abandoning a correct answer when a user pushes back. Queen and Zou suspect these errors share a correct outcome, which gives developers a clear target to train toward.

What survived: preferences, sycophancy and bias

Of the 11 findings that held or got stronger, three involve LLMs acting as evaluators. Models prefer their own answers. They also rate accepted and rejected papers similarly, and are swayed by linguistic spin in medical abstracts.

Three more involve sycophancy and agreement toward users.

Bias is a mixed picture. Left-leaning political bias persists. Gender stereotypes and implicit association biases have weakened. Biases against speakers of African American English have flipped direction in the trend data.

The citation test: 55 percent to zero, but not everywhere

As a case study, the pair looked at fabricated scientific citations. In the original 2023 study by Walters and Wilder, GPT-3.5 made up 55 percent of its references. Queen and Zou sent the same prompts to current models. GPT-6.1 Sol fabricated 0 percent, and Claude Sonnet 5.5 came in at 1 percent. Qwen3.7-Flash, also a 2026 model, still fabricated 69 percent of its references in the same test.

Citation fabrication is the cleanest case in the study. The original prompts could be repeated exactly.

The improvement is real but partial. On two-hop questions, a model must combine two known facts. Claude Sonnet 5.5 still fails in about 10 percent of cases where it correctly answers both individual questions. The original paper reported 43 percent for ChatGPT.

The gap between models is wide, and deployment is speeding up

Among models released between July and September 2026, expired findings range from 11 for DeepSeek V4 Pro to 30 for GPT-6.1 Sol. Size does not explain it. Release date, provider and model size do not reliably predict the pattern.

Meanwhile the systems these models sit inside are moving into production. Microsoft researchers reported that coding agents often overcome problems by using tools. They still struggle when they must understand unfamiliar codebases, according to Microsoft Research Focus published on 29 October. The same edition describes ForkPilot. That framework cut AI processing demands by up to 59 percent across six benchmarks.

Security work shows the same compression. OpenAI said on 27 October that Check Point had cut vulnerability research from hours to minutes using its Daybreak models. The claim comes from Check Point chief technology officer Jonathan Zanger.

The hardware bill is climbing with it. Hyperion Research reported on 3 November that on-premises HPC and AI-for-science revenue reached $16.5bn in the first half of 2026. That is up 19.9 percent from $13.8bn in the second half of 2025. Unit sales grew only 4.4 percent over the same period. That points to rising system costs. Dell took the market lead with $4.2 billion in server revenue in the first half of 2026. It ended more than 25 years of HPE leadership.

Earl Joseph, chief executive of Hyperion Research, put the demand down to two things. Organizations are replacing aging machines, he said. They also want to adopt AI into their workflows.

Put together, the two trends pull in opposite directions. Buyers spend more per system. At the same time, the published evidence they use to compare models ages out in about 15 months. A benchmark result from a year ago may describe a model that has since been replaced.

What the authors want, and what is still open

Queen and Zou conclude that empirical findings about LLMs should be presented as observations about specific models at a specific point, not as general truths. Their recommendation is narrow. Cite the model, cite the version, cite the month.

The authors warn against an all-clear.

The Reversal Curse and contradictory answers to paired time-related questions have weakened. They are still detectable. A claim that a system fails at some task is a statement about one build, not about the technology. Whether the pattern holds for models released after September 2026 is unknown.