Benchmarks are getting better at measuring AI doubt, not just AI answers

Two pieces of work attack the same problem from opposite sides. They did not land on the same day. GitHub published ReviewBench, an open benchmark for AI code review, built from an analysis of 103.9 million pull requests on its platform. Separately, UC Riverside computer scientists described a method for finding the internal features that make a language model sound confident while being wrong.

Neither is a product launch. Both try to answer a question buyers and users keep asking: when a model says something, how would anyone know whether to believe it?

What GitHub built

ReviewBench, announced on the GitHub Blog, contains 219 public pull requests from 187 open source repositories across 19 languages. GitHub says the language and repository-size mix closely matches its overall distributions. Senior engineers labelled the golden true positives by hand before release. The dataset is public.

The benchmark scores four metrics and labels every finding by severity and category. It lets teams slice results by the kind of review they care about. GitHub says the change helps it compare reviewers that surface many issues against ones that stay quiet, and strong catch rates against small suggestions.

The important number is 103.9 million. That is how many pull requests GitHub analysed to characterise real review workloads, and the 219 cases follow that distribution. GitHub made one deliberate adjustment. Pull request size is weighted toward the reviewable middle and tail, which reduces the overrepresentation of very large changes. The company also says its offline evaluation of Copilot code review now anticipates the direction of production experiments more reliably.

What remains unknown is how any competitor scores. GitHub invites other code review systems to onboard and submit results. The post does not report a leaderboard or a baseline model's numbers.

Why confidence is not correctness

The UC Riverside study, led by doctoral student Het Patel, looked inside two open-weight models: Meta's Llama-3.1-8B and Google's Gemma-2-9B. The team asked multiple-choice questions and sorted answers into four buckets, correct or incorrect crossed with confident or uncertain.

Using sparse autoencoders, they found three kinds of internal features: some tied mainly to uncertainty, some mainly to wrong answers, and some mixed. Suppressing features tied purely to uncertainty sharply cut accuracy. That suggests those features are doing real work in producing good answers. The university's write-up says altering some features changed model behaviour without retraining the whole model.

"The main assumption in the field is that when the model is confident, it is likely to be correct, and when the model is unsure, it is more likely to be incorrect," Patel said. "But we often see the counterexamples that are well documented."

That is the part worth holding on to. A confidence score is not a reliability score, and a model that hedges may still be right. Some teams build guardrails that block answers below a confidence threshold. The study implies that threshold is measuring the wrong signal.

Smaller models, measured claims

Quieter examples of the same discipline appeared elsewhere. A GitHub project called Jeff-Code pairs a 0.8B decision model with Qwen 3.8-27B inside a coding agent. It reports tasks finishing 47% faster (32% less time) at the same pass rate. The numbers are unusually precise: 62.4% versus 62.8% pass rate, a paired difference of minus 0.2 points with a 95% interval of minus 2.6 to plus 2.1, over 1,242 paired tasks.

The project also reports where it fails. There was no clear speed-up on Terminal-Bench 2.0 (0.96x) or SkillsBench (0.91x). Turning thinking off on every turn was faster but cost 7.6 points. A less cautious threshold of 0.7 cost 4.5 points.

Reflection, meanwhile, introduced Beam, a 501B-parameter open-weight model with 23B active parameters, and published a wide benchmark table alongside it. Beam scores 80.9 on SWE-bench Verified and 97.8 on AIME 2026. The company says it reaches results comparable to GLM-5.2 using 3 to 4 times less inference compute. The model is still in final red-teaming; weights, a technical report and a model card are promised later this month.

A vendor's own benchmark table is not neutral evidence, and Reflection's caveats matter. Its AA Omniscience index is marked public split only, with its own results. Still, the efficiency claim is the kind that can be checked once the weights ship.

Where the numbers get borrowed

Familiar benchmark names travel fast, and not always intact. Liquid AI's d1 decision model is compared against GPT-6.1 Sol and Claude Opus 5.5 on six applications. The company claims it matches or beats GPT-6.1 Sol on four and costs 19 to 200 times less. It reports 85% to 97% accuracy sorting good and defective parts in a visual inspection demo. It notes the model was never trained for that task.

Interfaze published a benchmark table for its Interfaze 1.1 Lite model. The table runs from GPQA Diamond (85.9, against 92.4 for Interfaze) to VoxPopuli speech recognition word error rate. It is a vendor comparison, not a third-party audit.

Outside the model vendors, Veris published a voice-agent benchmark that is more interesting for what it measures. On VAmoS Pro Bench, a television playing behind the caller cut pooled task completion from 38.7% to 8.6%. Nine of fourteen voice stacks completed fewer than 5% of TV calls. The top stack, Phonic, reached 57.5% task completion at 1.42 seconds of latency; the bottom, a Pipecat cascade, reached 17.3%.

Veris is candid about the limits. The board is one exam, built on one utility's policies, with simulated callers. That is a narrower claim than a general capability ranking, and Veris states it as such.

The older argument underneath

None of this settles the deeper question. David Chalmers' paper on whether a large language model could be conscious, revised in August 2024, argues that current models lack recurrent processing, a global workspace and unified agency. He concludes it is somewhat unlikely that today's models are conscious but that successors may be.

That is background, not news, and it does not bear directly on whether a code reviewer catches a bug. But it explains why the confidence problem is hard. A model that produces fluent, certain text has no separate faculty for knowing it is wrong. The UCR work suggests such a signal can be found and adjusted. GitHub's work suggests it can be scored.

What neither provides is a single number a buyer can put in a procurement document. ReviewBench has no published competitor results yet. The UCR method has been shown on two open-weight models, not on the closed systems most enterprises use. Beam's weights are not out. Until those gaps close, the honest answer to "is this model reliable" is still a set of narrow measurements, each with a stated scope.