Skip to content
World clockEU--:--UK--:--USA--:--CN--:--PLDEFRIT中文EN

portal about AI and technologyevents · analysis · interviews · technical background

Search
LIVE
›

Benchmarking AI reasoning is getting harder, and the numbers are getting stranger

A researcher ran 27,489 newly published CVEs through a local decision model. He found that 16.9% never state a security impact, a result that says as much about how AI systems grade text as about the bug reports themselves.

AI & modelsAnalysisGrace OkonkwoPublished: 4 October 20267 min readSources 8
Benchmarking AI reasoning is getting harder, and the numbers are getting stranger

The experiment is small and personal. Jerry Gamblin serves on the CVE Program's Quality, Automation and Consumer working groups. He ran Cloudflare's Clef models on a laptop against every CVE published between 4 August and 3 October 2026. He published the results on 4 October.

The headline finding: outside the Linux kernel, 989 of 23,736 descriptions, or 4.2%, never say what an attacker could do. Include the kernel, whose policy leaves impact to the user, and the figure rises to 4,650 of 27,489, or 16.9%.

Why it matters: this is what automated evaluation looks like when it leaves the lab. A 9B model answered two typed questions about each record in under a second. It needed no API key and no bill, per Gamblin's write-up. Anyone can now score a corpus at home. What is still unknown is whether those scores hold up outside a single researcher's setup. No one has tested the pipeline on a second machine or a second corpus.

Gamblin is careful about that. He checked the two-step pipeline against 129 CVEs he labeled by hand. It caught 15 of the 16 he judged as stating no impact. It never flagged one of his positive labels as negative. Clef 27B re-read everything Clef Flash marked as having no impact and overturned 1,198 of 2,187, or 54.8%. Flash alone, he writes, is too strict.

Why everyone is building a benchmark

The same week, OpenAI published an account of a coordinated effort to extract protected reasoning from its models. The activity began on 1 July, according to the company. It spiked on 24 and 25 July with 16,000 requests using an extraction pattern from more than 4,000 users. OpenAI says it disrupted a related cluster of more than 15,000 users by 28 July. It attributes a core cluster to individuals associated with Moonshot AI, the developer of Kimi.

That is a security incident, not a benchmark. But it shows the same pressure. Once model capability is measured, it becomes a target. The measurement itself becomes an attack surface.

OpenAI says the operators did not break encryption or reach stored conversations. They manipulated model interactions so hidden reasoning could be reproduced in visible form.

Benchmarks have become the public face of that capability. The volume of new evaluation work in the past few days is striking. Google opened an 85-task engineering benchmark and agent competition on 3 November, according to the headline list circulating that day. OpenAI introduced a LifeVal benchmark for everyday tasks the same day. Artificial Analysis overhauled its index to v5 on 29 October. It reported that agentic AI widened the proprietary-model edge to 12 points.

Each of those is a claim about how well machines reason. None of them settles the question. A benchmark is a sample. The sample is chosen by the company being measured or by a rival.

The numbers that get quoted

Take the reasoning scores that have dominated coverage. Startup Fortune reported on 4 October that OpenAI's GPT-6 Astra reached 62.7% on what it called AI's hardest reasoning test. It also reported that its ARC-AGI-3 score more than doubled the prior best. NewsBytes reported on 3 October that Google's Gemini 3.1 Pro scored 77.1% on ARC-AGI-2. Those are different tests with different ceilings, and the headlines rarely say so.

Then there is price and coding. Startup Fortune reported on 3 October that Claude Opus 5.5 was beating OpenAI's Astra on coding benchmarks and on price. SitePoint published an explainer the same day on what Claude Opus 5's FrontierCode score means at medium effort.

Sit those two together and you get a familiar problem. A model tuned to spend less compute scores differently from the same model at full effort. The leaderboard entry rarely carries the setting.

Artificial Analysis's v5 overhaul is an admission of the same thing. When the index changes, historical scores stop being comparable. The company said the gap between proprietary and open agentic models widened to 12 points under the new version. That is a real finding about the new version. It is not necessarily a finding about the models.

"Our results demonstrate that information theoretic approaches to collapse mitigation are efficient, and suggest new approaches for maintaining multi-agent diversity."

That line comes from a paper submitted to arXiv on 1 October by Lewis Mitchell, a single author at the University of Bristol, according to the listing. The work is about model collapse, not leaderboards, but it belongs in the same conversation. Mitchell ran a six-generation QLoRA fine-tuning experiment on Llama-3.1-8B. He found that filtering training text by an entropy rate estimate, computed from raw text with no model at all, produced 42% more unique trigrams, 30% more vocabulary and 19% less repetition than an unfiltered run. The logprob-based baseline, the established method, showed no significant benefit on any metric.

The lesson for benchmark watchers is uncomfortable. A metric can look rigorous and still measure the wrong thing. Mitchell's filter needed no model to score. That is exactly why it could be tested cheaply and repeatedly. Most reasoning benchmarks cannot say the same.

Local models as graders

Gamblin's setup is the clearest example of the shift. Ollama 0.35.1 added a decision endpoint that takes state and a set of typed questions. It returns choices, probabilities and scores, with no text to parse. Clef is a 27B model fine-tuned from Qwen 3.8. Clef Flash is a 9B model fine-tuned from Qwen 3.5. Both are Apache 2.0, with weights on Hugging Face, and both run locally from the Ollama library.

The Flash download is about 10 GB.

That changes who can evaluate what. A security team can score thousands of records overnight on a laptop. A solo researcher can reproduce the run. The catch is calibration. Gamblin needed the larger model to second-guess the smaller one, and even then he validated against a hand-labeled set of 129. At that scale, a few mislabels move the percentage.

He is upfront about the limits. The 4.2% figure excludes the kernel. The 16.9% figure includes it, and the kernel's own policy is the reason its records say nothing about impact. Neither number is a verdict on the CVE Program. Both measure how consistently writers fill in one field.

What the scores leave out

Consumer AI products are making the same trade. Google will restrict Gemini model access for personal accounts starting 9 October, according to an updated support document reported by 9to5Google on 3 October. Free users will be limited to Flash-Lite. AI Plus subscribers at $4.99 a month lose Pro and keep Flash-Lite and Flash. AI Pro at $19.99 and AI Ultra at $99.99 or $199.99 keep all three. THE DECODER reported the same tiering on 4 October. It noted that AI Pro customers also gain the Deep Think option.

Read that against the benchmark headlines and a pattern appears. The models being scored are increasingly the ones behind a paywall. A user comparing Gemini, ChatGPT and Claude on a public leaderboard may never touch the configuration that produced the score.

OpenAI's Dev Day announcements, reported by TechCrunch on 29 September, push further in the same direction. ChatGPT, which the company says has 1.2 billion weekly users, will suggest apps inside the conversation. 16 launch partners have signed on to Sign in with ChatGPT.

None of this makes benchmarks useless. It makes them partial. The most quoted numbers of the week, 62.7% on one reasoning test and 77.1% on another, describe specific runs under specific settings. The 16.9% CVE figure describes one researcher's laptop, one weekend and one pair of open models. All three are honest if you read the method. All three mislead if you read only the number.

What is missing is a shared standard for reporting the setting alongside the score. Neither Google nor OpenAI has said it will publish effort levels and tier access next to benchmark results. Until someone does, the comparisons will keep drifting.

Comments 0

Sources

8
  1. 01Decision Models vs. CVE DataEN
  2. 02Gemini app limiting what models free and AI Plus users can accessEN
  3. 03Google's new Gemini tiers cut free users to its weakest model and lock $5/month subscribers out of ProEN
  4. 04Disrupting a coordinated model-distillation campaignEN
  5. 05No Model Required: Text Entropy Rate Filtering Mitigates Iterative Fine-Tuning CollapseEN
  6. 06OpenAI's latest features take direct aim at the app store modelEN
  7. 07Anthropic asks Claude users to share voice data for AI model trainingEN
  8. 08Ideogram 4.5: The most precise edit modelEN

All figures and quotations in this text come from the sources listed below.

Content prepared by the editorial team with AI assistance.

Grace Okonkwo

Grace Okonkwo

AI, models and technology

Grace Okonkwo covers AI, models and technology for FLASH24, working from primary sources such as model cards, API documentation and benchmark papers rather than vendor summaries. She checks training data provenance, evaluation conditions and reported scores against the underlying datasets before any figure reaches print. She interviews researchers and engineers directly, tracks release calendars from major labs, and compares successive model versions on the same tests. Her own self-hosting, home-network and documentation-reading habits feed straight into that desk, since she tests tools on her own hardware first. She does not publish benchmark claims without a reproducible method.

Newsroom →

Comments

0
  1. No comments yet — be the first.

Write a comment

Comments are public. We do not publish abuse, spam or advertising.