Stanford study finds 44% of widely cited LLM claims have expired

The finding lands on anyone who cites a benchmark. Of the 50 claims Queen and Zou tested, 44 percent had already expired.

That matters for buyers, not just for authors. Anyone choosing a model, writing an internal AI policy or repeating a benchmark in a slide deck is probably working from evidence that is a year or more old. A number that was true in 2024 may say little about the model a team buys in 2026. The study also undercuts model-to-model comparisons: among models released between July and September 2026, the number of expired findings ran from 11 to 30. What nobody has published is a method for predicting in advance which claims will hold and which will rot. Neither release date, provider nor model size reliably predicted the pattern, the authors found.

The rest of the distribution is just as unflattering. Another 34 percent of the claims had weakened significantly, 16 percent still held and 6 percent had gotten stronger.

Fifty claims, twenty models, eight developers

Queen and Zou re-ran the 50 claims against 20 models released between July 2024 and September 2026 by eight different developers. Among the 22 findings that carried an estimated expiration date, the average lifespan was 15.7 months. Expiry accelerates. Sixteen percent of findings had expired after one year, and 39 percent after two.

Fifteen months is roughly one model generation.

The claims that rot fastest are about limits

The pattern is not random. Five of nine reasoning-related findings are counted as expired, and none of them held up or got stronger. The casualties include familiar complaints: models that miscount the letters in strawberry, that stumble on arithmetic in unfamiliar number systems, that lose information in the middle of a long context, that abandon a correct answer when a user pushes back. The researchers suspect these errors share a correct outcome, which gives developers a clear target to train toward.

Claims about behaviour last longer. Of the 11 findings that held or got stronger, three involve models acting as evaluators and three involve sycophancy and agreement toward users. Social biases cut both ways: the left-leaning political bias persists, while gender stereotypes and implicit association biases have weakened. Bias against speakers of African American English has flipped direction in the trend data.

A citation test with one large exception

The paper's case study is fabricated references. In the original 2023 work by Walters and Wilder, GPT-3.5 invented 55 percent of its citations. Sent the same prompts, GPT-6.1 Sol fabricated 0 percent and Claude Sonnet 5.5 fabricated 1 percent. Qwen3.7-Flash, also a 2026 model, still fabricated 69 percent.

Two 2026 models sit at opposite ends of that test.

The authors warn against reading it as an all-clear. On two-hop questions, Claude Sonnet 5.5 still fails in about 10 percent of cases where it answers both component questions correctly, against 43 percent reported for ChatGPT in the original paper. The Reversal Curse and contradictory answers to paired time-related questions have weakened, but both remain detectable.

Model-to-model spread stays wide even inside one release window. Among models shipped between July and September 2026, expired findings ranged from 11 for DeepSeek V4 Pro to 30 for GPT-6.1 Sol.

Other labs are answering with harder tests

Microsoft's research roundup for late October described two such efforts. CABRA generates coding challenges from scratch so complexity can be raised in a controlled way; across thousands of generated tasks, agents often solved problems by reaching for tools but still struggled to understand unfamiliar codebases. ForkPilot, a self-improving framework, matched the best existing methods across six benchmarks while cutting processing demands by up to 59 percent.

Elsewhere the answer is formal verification. An arXiv paper on entanglement-assisted quantum error-correcting codes used a researcher-in-the-loop pipeline with Lean proof checking, closing 54 gaps in archived tables and releasing 115 verified code records.

Its own numbers make the case. Validation-selected iterative programs reached 46.1 percent and 31.6 percent success on held-out executions at unseen lengths n=13 and n=15, against 19.5 percent and 3.1 percent for independently generated programs. The gap widens at the longer length.

Commercial labs are selling speed instead. Check Point CTO Jonathan Zanger said Daybreak cut vulnerability research from hours to minutes. The tool gives defenders access to OpenAI's frontier cyber models, which analyse vulnerabilities and trace attack paths.

What the authors want changed

Queen and Zou want the framing changed. They argue that empirical findings about LLMs should be presented as observations about specific models at a specific point, not as general truths. Papers should document what they tested, they add.

The practical instruction is blunter. Re-test before you cite, and date the test.