Reflection's Beam puts a 501B open model on efficiency, not raw scores

Beam is a sparse mixture-of-experts design trained on 23.8 trillion tokens. The reinforcement learning run used 10.5K Nvidia GB300 GPUs for four weeks and generated more than 100 million rollouts. Those figures come from Reflection's own announcement.

What that buys, per Reflection, is not a benchmark leader. It is a cheaper token. On Terminal Bench v2.1 Beam scores 80.1 against 88.3 for Kimi K3 and 88.2 for GLM 5.3. On SWE Bench Pro v2-Hard it posts 77.2, behind GLM 5.3 at 84.3 and Kimi K3 at 88.2. Reflection states plainly that frontier open models like Kimi K3 remain ahead on raw capability. For buyers, the pitch reframes the purchase decision: less about who tops the table, more about cost per finished task at volume. Reflection claims Beam reaches reasoning scores comparable to GLM-5.2 while using 3 to 4 times less inference compute, and says the gap widens against 2T-plus models such as Qwen 3.8-Max.

Nobody outside Reflection has verified any of it yet. The weights, technical report and model card are promised for later this month, and the company is taking early-access signups in the meantime. Benchmark figures published by the vendor before independent runs should be read as claims, not results.

The benchmark side is moving too

GitHub published ReviewBench, an open benchmark for AI code review built from an analysis of 103.9 million pull requests. The dataset itself is 219 pull requests from 187 public repositories across 19 languages, with every finding labelled for severity and category. GitHub says senior engineers validated the golden set independently, and that the benchmark now helps it predict whether Copilot code review changes will hold up in production.

Liquid AI used the same day to ship d1, its first decision model, which returns probabilities instead of generating tokens. Liquid says a text decision takes 200 to 300 milliseconds, that d1 matches or beats GPT-6.1 Sol on four of six applications tested, and that it costs 19 to 200 times less than either GPT-6.1 Sol or Claude Opus 5.5. Those comparisons, again, come from the vendor.

A third release went the other way, small rather than large. Jeff-Code, a fork of the Pi coding agent, pairs a 0.8B decision model with Qwen 3.8-27B. Its author reports a 62.4% pass rate against 62.8% for Qwen alone across 1,242 paired tasks, with time per task at 0.68 times the baseline. The 95% interval on pass rate runs from minus 2.6 to plus 2.1 points, so the quality difference is not established. The speed is the claim.

What the numbers do not settle

Interfaze published Interfaze 1.1 Lite, a mixture-of-architectures model that runs on a single 80 GB GPU. Its own table puts it at 85.9 on GPQA Diamond against 92.4 for Interfaze and 82.8 for GPT-5.4-Mini, and at 83.8 on olmOCR-Bench against 85.7 for Interfaze. A page of self-reported scores across five competitors is hard to audit, but the single-GPU footprint is a concrete deployment constraint readers can check.

Then there is the question of whether these benchmarks measure what buyers need. Veris ran 14 voice stacks across 2,694 simulated calls on VAmoS Pro Bench and found pooled task completion fell from 38.7% to 8.6% when a television played behind the caller. Nine of the 14 stacks completed fewer than 5% of those calls. Phonic led the board at 57.5% completion; Grok Voice, which the page headlines, sat second at 44.7%.

UC Riverside researchers offered a reason to distrust confident outputs at all. In a study published on 23 September, they examined Meta's Llama-3.1-8B and Google's Gemma-2-9B with sparse autoencoders and found features tied to confidence and features tied to correctness were separable. Suppressing uncertainty features sharply cut accuracy. Confidence, in other words, is not a proxy for being right.

Reflection's efficiency claim will get the same treatment once the weights land. Until then, the most useful thing about Beam is the shape of the argument: not that it wins, but that it costs less to lose by a little.