Open weights are not open source, and the gap to closed models is closing anyway

Two documents published within four days of each other in September describe the same market from opposite ends. Mozilla's report measures how close open-weight models have come to closed ones. The Open Source Initiative measures how much those weights actually let you do.

The numbers from Mozilla are the more quotable. Version 1.1 of its State of Open Source AI report, published Sept. 15 with data current to Sept. 1, found the best open model trailed the closed leader on the Artificial Analysis Intelligence Index by three points at 60% of the price, according to Tom's Hardware. Against Claude Fable 5 the gap was two points at 30% of the price. Mozilla's fit on METR task-horizon data puts the open-closed lag at roughly 4.4 months, in line with Epoch AI's four-month estimate.

What the four-month figure actually measures

METR scores a model by the length of task, in human working time, that it completes half the time. By Mozilla's fitted estimate, closed models handle tasks that take human experts 8 to 12 hours, and open models reach that point about four months later. On Mozilla's computation, open capability doubles every 3.9 months against 5.5 months for closed models.

Mozilla is the nonprofit behind Firefox and an advocate for open models. Its report is built on a Mozilla/SlashData survey of roughly 1,400 developers, OpenRouter traffic data and third-party benchmark indices. Tom's Hardware noted that TIME reported on July 14 that Mozilla CTO Raffi Krikorian described the report as partly advocacy. The report counts 16 notable open releases, but none delivers the data recipe required by the Open Source Initiative's definition.

"We see the decision to pay for closed [models] as workload-specific rather than organization-specific," Krikorian told Ars Technica in an email.

A second caveat sits in the report's own hardware chart. The best open model that fits on one server scores 52.6, and the best that fits on one GPU scores 40, drops of 10 and 23 points from the top. That is a wider gap than four months. Kimi K3's native MXFP4 checkpoint runs about 1.56TB across 96 shards, and Mozilla's serving configuration lists 64 or more accelerators, with vLLM calling for at least eight GB300 GPUs. The report describes this as open but not runnable by most who hold it. Thinking Machines' Inkling-Small, under Apache 2.0, fits one B300 at a 180GB floor.

Mozilla's data stops at Sept. 1, and the live boards have moved since. Artificial Analysis shifted its index to v4.3 with a different evaluation set, where Claude Fable 5.1 sits at 53 on its highest effort setting against Kimi K3 at 44. vals.ai's Terminal-Bench 2.1 board, updated Sept. 11, is led by GPT-6 Astra at 87.27%. Mozilla's own chart caption reads: "the gap resets every release cycle."

K3 also carries an allegation detailed in the Sept. 8 NSA/CISA/FBI joint advisory AA26-251A. The claim, which Mozilla's report states as "asserted, and unshown," is that Moonshot extracted Claude Fable 5 data to train K3 through distillation, the practice of training one model on another model's outputs.

The licensing argument underneath

The Open Source Initiative's Sept. 19 post takes the opposite tack. Open weights, it argues, let you exercise some of the four software freedoms, to use, study, modify and share, but not all of them, and only to a degree.

An AI model consists of three components: training data, weights and parameters, and code for data preparation and training. Open-weight releases publish only the middle one. That lets you run a model locally, pay a third party to host it, or fine-tune it on your own data. It does not let you fully modify or study the model, because the code and the training data, or a detailed account of how the data was built, stay private.

The OSI points to Ai2's Olmo as the counterexample. Because Olmo ships its full training dataset and model checkpoints, researchers could inject new information into the training data and watch how the model memorized or forgot it. That kind of study is only possible under a full open source release, the OSI argues.

The distinction is not academic. Mozilla counted eight of the top ten models by August token volume on OpenRouter as open weights, seven of them Chinese-built. Closed providers still took 96% of model-layer revenue on OpenRouter from May to September 2025, according to the Linux Foundation, which suggests buyers are weighing workloads rather than ideology.

Neither document settles the question its authors set out to answer. Mozilla's four-month figure is measured API to API on hosted endpoints at list price, and its own caption concedes the gap resets with every release. The OSI's argument is about what a license permits, not what a buyer needs. What both make clear is that "open" now covers a range of arrangements, and the label alone tells you very little about the model underneath it.