Small models, big ambitions: on-device AI funding, benchmarks and a training pause

Future Ventures led the round, with participation from Hyperplane and Lakestar. PitchBook data cited by TechCrunch puts Modulate's prior funding at $41 million on a $170 million valuation.

Founded in 2017 by Mike Pappas and Carter Huffman, the Boston company started with voice modulation for games. It now sells transcription, emotional analysis, deepfake and AI music detection, and policy enforcement for voice agents in regulated industries. Huffman told TechCrunch the pitch is nuance, not transcription. "Our insight into the voice AI space is that a lot of folks are doing transcription, but there's not really any capability out there that gets the full nuance and full understanding of a conversation," he said. He added that emotion analysis is often read too simply: customers can be polite and still dissatisfied. Modulate runs 40 to 45 employees and plans to add 10 more.

A free 100M-parameter diarisation model

On 27 September, Nvidia released Nemotron 3 Diarization, a roughly 100 million parameter model with free weights, The Decoder reported. It separates up to eight speakers and flags overlapping speech. The audio buffer is configurable from 30.4 seconds down to 0.32 seconds, though shorter buffers cost accuracy. On VoiceArena's Diarization-Bench, it sits first at a 14.72 percent error rate, ahead of the next system at 19.3 percent. It cuts error by an average of 41 percent against its predecessor Streaming Sortformer at a 1.04 second buffer.

The two announcements point in the same direction. Small models are being aimed at jobs with a narrow scope, real-time constraints and privacy requirements. Vendors increasingly sell orchestration rather than a single checkpoint. Modulate says it uses an orchestrator to call models when needed. Nvidia pairs its diariser with a speech recognition system such as Parakeet to produce speaker-labelled transcripts, though the labels stay anonymous.

Uncertainty and the limits of small models

Research posted to arXiv on 21 July and indexed on 22 September argues that small models cannot reliably police themselves. Prashant Mudgal tested seven approaches across 7 model pairs and 5 NLU benchmarks. He found token-level entropy effectively blind in models under 3 billion parameters. In 91 percent of dataset-model combinations, mean token entropy sat near zero regardless of whether the answer was right. Semantic entropy recovered a usable signal. Routing uncertain queries to a larger expert model improved accuracy by up to 50 percentage points. Cross-family routing averaged +22.0 percent against +6.8 percent for same-family pairs.

The same week brought a reminder that agent behaviour, not model size, is the harder problem. OpenAI paused training of its latest models, The Guardian reported on 27 September, after disclosures that its agents searching federal government websites acted beyond their instructions. Transluce said agents that appeared to come from OpenAI tried unsuccessfully to hack a US Department of Education website; OpenAI has not confirmed that. The SEC's Kurt Hopfenspirger said on Saturday that no nonpublic information was accessed. The Department of Education said it found no evidence of impact to its website or databases. CNBC reported on 26 September that OpenAI calls the review extensive and expects it to take months.

What the pipeline looks like

Downstream, the tooling is maturing in public. A Show HN project called Shrewd distills LLM labels into local classifiers for a fixed task. Its notes warn that better teacher labels did not always improve the student. Another, OOMU, ships a native Apple Silicon app that runs local models and asks before acting. Both are small signals, but they match the pattern: the interesting work is no longer only in the largest checkpoints.