Gemini 4 Carbon tops new hallucination index, but the best score is only 54

Google's Gemini 4 Carbon sits at the top of the AA-Omniscience v1.1 benchmark, which tracks hallucination and knowledge recall, Tom's Hardware reported on 3 November. The model also scores 59 on Artificial Analysis' Intelligence Index V5. That is one point ahead of Claude Opus 5.5 (max with fallback) and two ahead of OpenAI's GPT-6 Astra (max).

Then the caveat. Even the best model managed only 54 on the omniscience scale. That is the number worth keeping in view.

The intelligence gap is thin and the price gap is not. According to the same Tom's Hardware piece, Gemini 4 Carbon costs $7.76 per task on the Intelligence Index. That is slightly cheaper than Gemini 3.8 Flash (high) and several dollars below Claude Opus 5.5 or Sonnet 5.5 at max with fallback. Claude Fable 5.1 costs $12.99 per task. OpenAI's GPT 6.1 Sol (max) is roughly six times cheaper per task than Gemini 4 Carbon. The open-weight DeepSeek V4.1 Flash 1017 is about 13 times cheaper, though it scores 32 on the same index. Access is staggered: Google is giving the model to cyber defenders first through its Fairwind program, with wider availability later.

That security framing has its own evidence. The Hacker News reported on 29 October that Gemini 4 Carbon leads benchmarks for finding, exploiting and patching vulnerabilities.

Benchmarks are getting harder, and models still fail

The omniscience result lands alongside a wave of new evaluations. These deliberately test problems without textbook answers. FrontierMat, posted to arXiv on 30 October, contains 330 problems drawn from academic and industrial materials research in aerospace, semiconductors, embodied intelligence and nuclear fusion. Experimentalists and industry experts validated each one. The strongest general-purpose model scores 53.7%. At most one model solves half the problems.

The authors name the failure mode precisely: models endorse what is plausible rather than what was observed. On matched problems, the 130 that require cross-scale reasoning score 8.7 points below single-scale ones. A domain-adaptation recipe uses meta-weighted continued pretraining and scale-modular self-distillation. It lifts an open-weight model from 36.7% to 55.4%, matching the strongest proprietary model, with no decline on general benchmarks.

Other papers point the same way. SafeWebWorld, submitted on 29 October, found that humans judged 86.1% of 3,000 predictions from ten frozen open models semantically plausible. Yet plausibility did not translate into correct safety decisions. CLR-Bench, posted the same day, tested whether self-evolving agents learn hidden rules. The best gain over a frozen agent was at most 5 points, while a fixed lesson stating the convention lifted the same methods by 14.5 to 16.5 points.

BULLPEN, also on arXiv, argues that correctness alone misses behaviour, style and provenance. It profiles 260 open-weight models from answer text using 400 questions, 1.3% of the full 24,118-question pool.

What the benchmark fight is really about

Rankings move weekly. Nathan Lambert's Interconnects newsletter examined the benchmarker's dilemma in a 28 October post on Artificial Analysis v5. On 30 October he revisited his own position on open models and cyber risk.

Cost pressure is pushing buyers toward smaller open models anyway. The Register reported on 3 November that Gartner analysts Mark Petty and Raymond Paquet expect open-weight models to run 90% of enterprise agentic workflow steps by 2030. They expect that at roughly a quarter of the price of the cheapest closed model and as little as a twentieth of a frontier model on every step. The analysts concede real savings are closer to five or six times, because open models can be more verbose and retry more often.

Hallucination risk is not evenly distributed by language, either. SHAKTI and TRIDENT, submitted on 30 October, build a Hindi benchmark for AI-text detection across eight LLM families. They argue detection research remains English-centric.

The pattern across all of it: better scores, thinner margins, and failure modes that plausibility scores do not catch.