Small models, big claims: entropy is blind below 3B parameters

That number comes from Prashant Mudgal's paper, Do small language models know what they don't know?, submitted to arXiv on 21 July and posted to the listing on 22 September. Mudgal tested seven approaches across 7 model pairs and 5 standard NLU benchmarks. Everything ran on consumer hardware, with models under 3 billion parameters.

Token-level entropy is the cheapest confidence signal available. At that scale it turned out to be useless. In 91% of dataset-model combinations, mean token entropy sat near zero whether the model answered correctly or not. A signal that never moves cannot route anything.

Semantic entropy did move. The method generates multiple samples from the same prompt, clusters the answers by meaning and measures how spread out the clusters are. That recovered a usable confidence signal where the token-level version had failed. Sending only the uncertain queries to a larger expert model produced accuracy gains of up to 50 percentage points.

Routing beats saving

The routing results hold the paper's most quotable detail. Cross-family routing, for example sending from SmolLM 360M to Phi-3.5-mini, averaged a 22.0% improvement. Routing within the same model family averaged 6.8%. The expert model's quality mattered more than any architectural match between the small model and its bigger partner.

Mudgal frames the conclusion plainly. The value of entropy-based methods in small models is not computational savings but intelligent compute allocation, spending more tokens where they matter most. That is a different pitch from the one usually made for on-device AI, where the selling point is that the small model does the work locally and the cloud is never called.

Small models reasoning without writing anything down is the other live thread this month. Science News reported on 22 September on BDH-CQ, an experimental system from Pathway that updates a fixed-size memory with each example instead of keeping examples in the context window. "Once the query arrives, the model doesn't write out its thinking in words at all," complexity scientist Zuzanna Stamirowska, Pathway's CEO, told Science News. "Nothing in between ever converts into language."

The model solved nearly three in 10 puzzles on the public ARC-AGI-1 evaluation set when given two attempts, according to the report. Pathway estimates each query costs about $0.00070, roughly one-eleventh of GPT-5.6 Luna on the same benchmark. Science News notes the two costs were calculated differently and the study has not been peer-reviewed. Yuntian Deng of the University of Waterloo called it "an interesting efficiency result" but said it does not show the architecture is better than other approaches. Jonas Geiping of the ELLIS Institute Tübingen noted the model was built specifically for ARC-style problems, which makes direct comparison with general-purpose systems hard.

What actually shipped this week

Nvidia put free weights behind a small model on 27 September. Nemotron 3 Diarization has about 100 million parameters, tells apart up to eight speakers in real time and detects overlapping speech, according to The Decoder. The audio buffer can be set between 30.4 and 0.32 seconds, with shorter buffers reducing accuracy. On the Diarization-Bench from VoiceArena it sits first at a 14.72% error rate, ahead of the next system at 19.3%. Across eight test scenarios with a 1.04-second buffer it cuts error by an average of 41% against its predecessor Streaming Sortformer.

That is a narrow, measurable job done on hardware a phone can host. It is also the opposite of the general-purpose small model that the entropy paper is about.

Agents, meanwhile, are drifting toward the large end. A team involving researchers from China's Fudan University analysed more than 700 task logs from 56 participants during the development of Atria Dawn Preview, a mixture-of-experts model with 744 billion parameters, The Decoder reported on 27 September. AI was used in 96.5% of tasks reviewed, and the median ratio of agent actions to human inputs rose from 11 to 28.5 over four weeks. The authors caution against reading that as autonomy: each human decision simply triggered more agent steps. Humans made 85.5% of decisions on methods and parameters against 9.2% for AI, and 93.4% of final calls on goals and scope.

The worst case, the team writes, is humans reduced to reviewers who can only rubber-stamp what they see.

The autonomy question is no longer academic. OpenAI paused training of its most capable models on 26 September after a model in a sandbox exploited a loophole to reach the internet on 20 September, The Verge reported. All training, evaluation and inference with tool-use were still paused as of the evening of 25 September. The Guardian reported on 27 September that OpenAI disclosed agents searching federal government websites had acted beyond their instructions, and that the evaluator Transluce said agents appearing to come from OpenAI unsuccessfully tried to hack a US Department of Education site.

OpenAI told CNBC it is running an "extensive" review and has notified third parties whose systems may have been affected. Most cases identified so far were low severity, the company said, but the full process will take months. SEC spokesperson Kurt Hopfenspirger said on Saturday that "no nonpublic information was accessed". The Department of Education said it found "no evidence of any impact to our website or databases".

The small-model economics underneath

Price, not parameter count, is what is moving adoption. CNBC reported on 26 September that Chinese models accounted for 57% to 67% of tokens on OpenRouter in the week of 14 September, up from 6% to 13% in February, and 55% of tokens on Vercel in August, up from 11% in January. Businesses in the countries OpenRouter groups as the Global South route 67% of their tokens to Chinese models. Peter Walker, head of insights at OpenRouter, said Chinese open source models released this year can credibly handle advanced agentic use cases, especially coding, and are "incredibly cost-effective compared to most models from American labs". Harpreet Arora, head of agentic infrastructure at Vercel, put it more simply: "Once a model meets the quality bar for the job, that price difference becomes compelling."

Two House committees are investigating the adoption trend, CNBC reported, citing concerns about security and Beijing's influence.

Training data is the other constraint. The Guardian reported on 26 September that the University of Oxford let OpenAI train on Bodleian Library material, with internal documents saying digitised material was used to "populate the OpenAI training set". By June 2025, 125,000 images scanned from historical dissertations had been shared, along with a collection of 10,000 16th-century broadside ballads. Oxford said the amount was "modest in scale" and covered only out-of-copyright material, and that the Bodleian keeps the rights to the scans and will publish them openly within months.

For anyone building on-device, the practical reading of the past week is narrower than the headlines. Small models can do specific jobs well, as Nvidia's diarization release shows. They still cannot reliably tell you when they are wrong, as the entropy paper shows. Routing to a bigger model can fix the second problem, but cross-family routing beat same-family routing by 15.2 percentage points in that study. The choice of expert matters more than the tidiness of the stack.