New benchmarks and studies take aim at AI models that sound sure and get it wrong

On 5 October, Reflection introduced Beam, its first open-weight model. It is a sparse mixture-of-experts with 501 billion total parameters and 23 billion active, pretrained on 23.8 trillion tokens. The company's blog says the weights, technical report and model card will follow "later this month." It also says Beam is still undergoing final red-teaming.

Reflection's own numbers show where it stands. On SWE Bench Pro v2-Hard, Beam scores 77.2 against 88.2 for Kimi K3. On Terminal Bench v2.1 it gets 80.1, while K3 reaches 88.3 and DeepSeek V4.1 Flash 90.6. The pitch is not primacy but cost. Beam claims reasoning scores comparable to GLM-5.2 using 3 to 4 times less inference compute, per the same post.

Why a confident answer is not a correct one

The evaluation problem underneath those tables has a research angle. A UC Riverside team, led by doctoral student Het Patel, examined Meta's Llama-3.1-8B and Google's Gemma-2-9B with sparse autoencoders. It sorted multiple-choice answers into four groups by whether they were right or wrong and whether the model was confident or uncertain. The team found features tied mainly to uncertainty, features tied mainly to wrong answers, and overlapping ones. Suppressing the purely uncertainty-linked features sharply cut accuracy, according to the university's write-up.

"The main assumption in the field is that when the model is confident, it is likely to be correct, and when the model is unsure, it is more likely to be incorrect," Patel said in the UCR release. "But we often see the counterexamples that are well documented."

The practical claim is that some of those internal features can be dialled down without retraining the whole model. Patel compared this to turning knobs. That matters because the usual fix for an overconfident model is another training run. The usual measure of an overconfident model is a benchmark that scores answers, not the model's own sense of doubt.

Benchmarks are getting built for the doubt

GitHub used 5 October to publish ReviewBench, an open benchmark for AI code review. It is built from an analysis of 103.9 million pull requests on GitHub. It contains 219 pull requests from 187 public open source repositories across 19 languages. GitHub says senior engineers independently labelled the golden true-positives before release. It also says every finding carries a severity and category label, so teams can slice results by precision and recall preferences.

The awkward admission in GitHub's post is about its own product. The company says that with ReviewBench, its offline evaluation of Copilot code review "has become more effective at anticipating the direction of production experiments." In other words, the old offline signal was not tracking what happened in production closely enough. A benchmark that predicts the real world is a different artefact from one that ranks models on a leaderboard.

Veris took the same instinct to voice agents. Its VAmoS Pro Bench, published on 5 October, ran 14 voice stacks across 2,694 simulated utility billing calls. Task completion ranged from 57.5% for the top stack to 17.3% at the bottom. No single headline number would have captured that spread. The board also recorded barge-in rates and cost per call. It found that a television playing behind the caller cut pooled task completion from 38.7% to 8.6%, while street noise at the same level cost nothing.

Small models, small decisions, measurable results

Not every evaluation needs a lab. A GitHub project called Jeff-Code, posted on 5 October, inserts a 0.8B decision model into the agent loop of Pi, Mario Zechner's coding agent, alongside Qwen 3.8-27B. Jeff makes two quick calls around each Qwen turn, roughly 0.2 seconds each. One asks whether it can take the next information step itself. The other asks whether Qwen needs to think hard.

The paired results, run across 1,242 tasks, put pass rate at 62.4% against 62.8% for Qwen alone. The authors give the difference as minus 0.2 points, with a 95% interval of minus 2.6 to plus 2.1. Time per task fell to 0.68 times the baseline, with a median of 0.70. Turning thinking off on every turn was faster still but cost 7.6 points. A less cautious threshold of 0.7 cost 4.5 points. The repository also notes that in about 5% of tasks the setup ran more than 30 minutes longer than Qwen alone.

That last detail is the kind of thing aggregate speedups hide. The same is true of the benchmark design argument Matklad made on 5 October. Pick an input size that makes a micro-benchmark run about 300ms. Anything under 10ms risks being skewed by fixed costs like interpreter startup, and anything over a second makes repeated runs slow. His stated premise is that benchmarking exists to give the author enough intuition to make a correct decision, not to produce a precise measurement.

A wave of decision models, each with its own scoreboard

Liquid AI used 5 October to launch d1, a decision model that returns probabilities rather than generating tokens. It now accepts images as well as text. The company says it tested d1 against GPT-6.1 Sol and Claude Opus 5.5 on six applications. It matched or beat GPT-6.1 Sol on four, at 19 to 200 times lower cost. On a public industrial inspection dataset, it sorts good and defective parts with 85% to 97% accuracy, according to the blog, without having been trained for the task.

Interfaze published a different shape of system the same day: Interfaze 1.1 Lite. It mixes a reasoning core with specialist models for documents, speech, segmentation and forecasting, and runs on one 80GB GPU. Its published table puts it at 85.9 on GPQA Diamond against 92.4 for Interfaze and 89.9 for Claude-Sonnet-4.6. It puts it at 83.8 on olmOCR-Bench against 85.7 for the larger Interfaze model. These are vendor-reported numbers from vendor-chosen benchmarks, which is exactly the measurement gap the code review and voice benchmarks are trying to close.

The counterweight is cost and speed. A model that answers in 200 to 300 milliseconds with a probability can be checked or discarded cheaply. A model that generates a paragraph of confident prose cannot. Sean Goedecke argued on 5 October that decision models like Jev will be easy to distil into hand-built classifiers once a use case is validated. The prompt and its outputs give you labelled data for free. The trainer becomes the trainee.

What the evidence outside AI looks like

For contrast, consider what a large, well-anchored study looks like. PsyPost reported on 4 October on a study of more than 2.5 million children, published in The Pediatric Infectious Disease Journal. It found no link between MMR vaccination before age two and autism. The lead researcher, Todd Burstain of UW Medicine, told PsyPost that earlier European studies drew on 2000-2012 data, when autism prevalence was about a third of current US levels. He said an age-anchored landmark analysis avoids immortal time bias.

That is the standard AI benchmarks are being measured against, and most of them are not close. Vendor tables are self-reported, often on benchmarks the vendor selected, sometimes with rows marked "not reported" for competitors. Reflection's Beam post is unusually candid about this, listing rivals' missing scores alongside its own, but the missing entries are still missing. GitHub's ReviewBench, Veris's voice board and the Jeff-Code paired runs all move in the same direction: publish the harness, the distribution and the interval, not just the headline.

Other releases on 5 October fit the pattern of small, measurable tools rather than grand claims. Reika, a coding agent CLI built around 8B to 35B local models, publishes its own findings. One mid-context rewrite reprocessed 8,453 tokens of prompt, costing 7.9 minutes of prefill. An append in the same session cost 25 tokens and 3.5 seconds. Precogly shipped version 0.4.0 of an open-source threat modelling platform and joined OWASP. NASA and IBM published fine-tuning code for a lunar foundation model covering crater detection and ice prospectivity.

None of these settle whether a model knows what it does not know. The UCR work suggests the internal signal exists and can be adjusted. The new benchmarks suggest the external signal has been too coarse to notice. Both are ways of saying the same thing. An answer and an assurance are two different outputs, and only one of them has been measured properly so far.