Open weights are closing on frontier models, but the label is doing too much work

Mozilla published version 1.1 of its State of Open Source AI report on Sept. 15, using data current to Sept. 1. Tom's Hardware reported on Sept. 16 that the best open model trailed the closed leader on the Artificial Analysis Intelligence Index by three points at 60% of the price, and sat two points behind Claude Fable 5 at 30%. Mozilla's fit on METR task-horizon data puts the open-closed gap at around 4.4 months. That lines up with Epoch AI's four-month estimate.

The methodology matters, because the headline number travels further than its caveats.

METR scores models by the length of task, measured in human working time, that they complete half the time. By Mozilla's fitted estimate, closed models handle tasks that take human experts 8 to 12 hours. Open models reach that about four months later. Open capability doubles every 3.9 months, compared with 5.5 for closed, according to Mozilla's computation. The report also charted vals.ai's Terminal-Bench 2.1 results, which run every model through the same harness, the software layer that offers a model its tools.

On that board, Z.ai's GLM-5.2 scored within a point of Claude Opus 4.7 and about four points behind Opus 4.8, at less than one-fifth the cost per test. On OpenRouter, a marketplace that routes developer traffic to hundreds of models, Mozilla counted eight of the top ten models by August token volume as open weights, seven of them Chinese-built.

Then the money.

Closed providers took 96% of model-layer revenue on OpenRouter from May to September 2025, according to the Linux Foundation. "We see the decision to pay for closed [models] as workload-specific rather than organization-specific," Mozilla chief technology officer Raffi Krikorian told Ars Technica in an email, as quoted in the Tom's Hardware report.

What the four-month figure leaves out

Two caveats sit inside the report itself. The four-month gap and the 30% token price figure are measured API to API on hosted endpoints, at list price. The report's own hardware chart puts the best open model that fits one server at 52.6 on the index and the best on one GPU at 40, drops of 10 and 23 points from the top. That is a wider gap than four months.

The serving requirements are the practical version of the same problem. Kimi K3's native MXFP4 checkpoint runs about 1.56TB across 96 shards. Mozilla's serving configuration lists 64 or more accelerators, while vLLM calls for at least eight GB300 GPUs, with multiple nodes for production traffic. The report describes this as open but not runnable by most who hold it. Tom's Hardware put the memory need near 1.5TB in July. One exception the report names is Thinking Machines' Inkling-Small, under the Apache 2.0 license, whose NVFP4 version fits one B300 at a 180GB floor.

The report's data stops at Sept. 1, and the boards have moved since.

Artificial Analysis shifted its index to v4.3 with a different evaluation set. The live board has Claude Fable 5.1 at 53 on its highest effort setting with Kimi K3 at 44, numbers that are not comparable to the v4.1.1 figures Mozilla plotted. vals.ai's Terminal-Bench 2.1 board, updated Sept. 11, is now led by GPT-6 Astra at 87.27% with Fable 5.1 at 85.02%. Mozilla's own chart caption reads: "the gap resets every release cycle." K3 also carries an allegation detailed in the Sept. 8 NSA/CISA/FBI joint advisory (AA26-251A). The claim, which Mozilla's report states as "asserted, and unshown," is that Moonshot extracted Claude Fable 5 data to train K3 through distillation, the practice of training one model on another model's outputs.

The naming fight underneath the benchmark fight

The Open Source Initiative used the same news cycle to argue about vocabulary. In a Sept. 19 post, OSI wrote that open weights let you exercise some of the four software freedoms, the freedom to use, study, modify and share without asking the rights holder, but not all of them, and only to a certain degree.

OSI's framing: an AI model has three main components, training data, weights and parameters, and code for data preparation and training. Open-weight releases ship the weights, which lets you run a model locally, pay a third party to host it, or fine-tune it on your own data. They do not ship the other components. Without the code and the training data or an account of how the data was built, OSI argues, you cannot fully inspect a model to answer why it produced an output or whether that output can be trusted. The post cites Ai2's Olmo as an example of a large language model whose full training dataset and checkpoints let researchers study how a model memorizes or forgets injected information.

The distinction is not academic for anyone reading the Mozilla numbers as a buying guide. Mozilla's report counts 16 notable open releases, but notes that none delivers the data recipe required by the Open Source Initiative's definition. Mozilla is also an advocate for open models: TIME reported on July 14 that Krikorian described the report as partly advocacy. The report is built on a Mozilla/SlashData survey of roughly 1,400 developers, OpenRouter traffic data and third-party benchmark indices.

What none of these documents settle is whether cheaper, slightly-behind weights are enough for the workloads people actually run. The revenue split says most buyers still pay for closed. The token volume split says most traffic is already open. Both can be true at once, and the four-month figure does not decide between them.