Benchmarks are everywhere, but they measure less than the labs claim

GitHub published ReviewBench on 5 October. It is an open benchmark for AI code review, built from 219 public pull requests across 19 languages. The company says it modelled the set on 103.9 million pull requests that have passed through GitHub. Senior engineers labelled the true positives before release. GitHub says the dataset is public and ready to onboard new code review systems.

The timing matters less than the construction. ReviewBench scores reviewers on four metrics. It labels every finding by severity and category, so a team can see whether a model catches critical bugs or just produces noise. GitHub also says its offline evaluation of Copilot code review now better predicts whether a change will help in production. That is the claim worth testing: a benchmark only earns trust when its scores track what users see.

A voice agent that fails when the TV is on

Veris published its VAmoS Pro bench results the same day, with a finding that reads like a warning label. The test ran 2,694 simulated phone calls for a utility billing task. A television playing behind the caller cut pooled task completion from 38.7% to 8.6%. Street noise at the same level cost nothing, according to the leaderboard. Nine of fourteen voice stacks completed fewer than 5% of the TV calls.

Veris ranks Phonic first at 57.5% task completion with a 1.42 second latency. Grok Voice Think Fast 2.0 follows at 44.7% and Gemini 3.8 Live at 40.1%. Hugging Face's Whisper, gpt-oss-120b and Kokoro stack sits last at 19.0%. Veris is explicit that this is one exam on one utility's policies. Treat the ordering as a snapshot, not a verdict on the vendors.

The same board tested eight decision models on when an agent should speak. Cloudflare's Clef Flash and Nimble talked over callers least. Clef Flash replied about a second sooner than Nimble. In a separate 600-call run, Veris said interruptions fell from 52% of caller turns to 11% with semantic turn detection. That number comes from the vendor's own testing, not an independent audit.

Efficiency claims need the same scrutiny

Reflection introduced its first open-weight model, Beam, on 5 October. It is a sparse mixture-of-experts design with 501 billion total parameters and 23 billion active. The company says it pretrained on 23.8 trillion tokens. It ran reinforcement learning across 10.5K Nvidia GB300 GPUs for four weeks. It also says Beam matches GLM 5.2 on advanced reasoning while using 3 to 4 times less inference compute.

Beam's own table undercuts the headline. On SWE Bench Pro v2-Hard, Kimi K3 scores 88.2 against Beam's 77.2. On Terminal Bench v2.1, Kimi K3 reaches 88.3 against 80.1. Reflection frames this honestly as an efficiency advantage rather than a capability lead. The weights, technical report and model card are still to come later this month.

Smaller projects make smaller claims. A fork of the Pi coding agent called Jeff-Code pairs a 0.8B decision model with Qwen 3.8-27B. It reports a 62.4% pass rate against 62.8% for Qwen alone, a paired difference of minus 0.2 points. It also cuts average time per task to 0.68 times. Liquid AI's d1 decision model launched on 5 October with image support. It answers in 200 to 300 ms without generating tokens, the company says. It costs 19 to 200 times less than GPT-6.1 Sol and Claude Opus 5.5.

Two things are still unknown. Nobody outside the labs has run these systems on tasks they did not choose. The vendors have not published the failure cases behind their averages. Until then, a benchmark score is a starting point for a buying decision, not the decision.