Stanford audit finds 44 percent of LLM claims expired as lab automation money flows

Labs buying automation are being asked to build workflows on models whose documented behaviour can change within about a year. Of the 50 claims Queen and Zou tested, 44 percent had expired, 16 percent still held and 6 percent had got stronger. Validation done at purchase may not describe the system six months later. The practical question for buyers is narrow: which model version was a claim measured on, and when.

What expired, and how fast

The two researchers tested 50 widely cited claims about large language models against 20 models released between July 2024 and September 2026 by eight developers.

Another 34 percent of the claims had weakened significantly, the study found. Among the 22 findings carrying an estimated expiration date, the average lifespan was 15.7 months. Expiry accelerates with time: after one year 16 percent had expired, and after two years the figure climbed to 39 percent.

Reasoning claims age worst. Five of nine reasoning-related findings are expired, and none held up or got stronger. The list includes counting letters in "strawberry," arithmetic in unfamiliar number systems, losing information mid-context, and abandoning a correct answer when a user pushes back. The researchers suspect these errors have a "correct" outcome, which gives developers a clear target to train toward.

Claims about preferences and tendencies last longer. Of the 11 findings that held or got stronger, three involve models acting as evaluators and three involve sycophancy toward users. Social biases are mixed: the left-leaning political bias persists, while gender stereotypes and implicit association biases have weakened, and bias against speakers of African American English has flipped direction in the trend data.

The money moving into automated labs

Opentrons Labworks said it raised $40 million in Series E financing led by Avego, with BroadOak Capital Partners participating. It did not disclose its valuation or the ownership stakes investors took. The AI Insider reported the round on 30 October.

The installed base is the pitch. Opentrons says it has deployed more than 11,000 robots across labs in over 80 countries, including at 90 percent of the world's 20 largest pharmaceutical companies. Founded in 2014, it sells the Flex platform, which pairs robotic instruments with software for standardising lab operations.

Proceeds go to AI-driven experimental execution, deeper integration with third-party instruments and software, and entry into additional markets, the company said. Its open, API-driven architecture is meant to slot into existing research environments rather than replace them.

"Millions of scientists spend much of their day on manual work a robot can do," CEO James Atwood said in the funding announcement. Avego managing partner James Flexner framed the bet the same way: "Every AI-driven discovery still has to be tested in a real lab".

The software layer is going open source

OpenScience is an open-source research agent that runs as a desktop app, a browser workspace or a terminal command, with shell, Python and R kernels. It ships connectors to ChEMBL, UniProt, PubMed and arXiv, plus ten bring-your-own-key adapters for NVIDIA BioNeMo NIM endpoints.

The project's own documentation is blunt about limits. Every step is visible, it says, and users should review sources, assumptions, code and outputs before relying on a scientific conclusion. The BioNeMo adapters are marked experimental and need the user's own NVIDIA API key.

Where the models still fail

Fabricated citations show how uneven progress is. In a 2023 study by Walters and Wilder, GPT-3.5 made up 55 percent of its references. On the same prompts, GPT-6.1 Sol fabricated 0 percent and Claude Sonnet 5.5 fabricated 1 percent. Qwen3.7-Flash, also a 2026 model, still fabricated 69 percent.

Two-hop questions, where a model must combine two facts it knows, remain a weak spot. Claude Sonnet 5.5 still fails in about 10 percent of cases where it answers both individual questions correctly, against 43 percent reported for ChatGPT in the original paper.

Model choice matters as much as model generation. Among models released between July and September 2026, expired findings ranged from 11 for DeepSeek V4 Pro to 30 for GPT-6.1 Sol. Neither release date, provider nor model size reliably predicts the pattern, the researchers found.

What the forecasts assume

The market forecasts assume the tooling keeps improving. The Business Research Company projects the AI in life sciences market rising from $3.59 billion in 2025 to $4.44 billion in 2026, which the release describes as a 23.6 percent compound annual growth rate. It forecasts $10.43 billion by 2030 at a 23.8 percent CAGR, citing AI-powered lab automation among the drivers.

That forecast covers AI across life sciences, from remote patient monitoring to drug discovery, not lab robots alone.

Queen and Zou draw a narrower conclusion than the forecasts. Empirical findings about LLMs should be presented as observations about specific models at a specific point, not as general truths, they argue. For a lab standardising on a robot, that means re-running the validation, not inheriting it.