Skip to content
World clockEU--:--UK--:--USA--:--CN--:--PLDEFRIT中文EN

portal about AI and technologyevents · analysis · interviews · technical background

Search
LIVE
›

Small models move on-device as Aleph Alpha ships Kolibri and llama.cpp adds decision models

Aleph Alpha released Kolibri on 3 October, an open-weight English-German model with 78B total parameters and just 3B active, downloadable under Apache 2.0. That same day llama.cpp added support for decision models that answer a query in a single forward pass.

AI & modelsAnalysisRachel NwosuPublished: 3 October 20265 min readSources 8
Small models move on-device as Aleph Alpha ships Kolibri and llama.cpp adds decision models

Aleph Alpha published Kolibri on 3 October, the Day of German Reunification, and framed the release around sovereignty rather than raw scale. The company describes it as an English-German Mixture-of-Experts Transformer with 78B total parameters and 3B active, supporting context lengths of up to 1M tokens. Full weights are on Hugging Face under Apache 2.0.

The pitch is deployment cost. Aleph Alpha says the model sits on the Pareto frontier for quality versus serving cost in both English and German, and that it matches models with up to four times its active parameter count, naming Nemotron 3 Super. The company says customers can run it on-premise without sending internal data to third-party inference services. That matters for buyers in regulated sectors. Public administration, industrials and aerospace are the target markets named in the announcement. Aleph Alpha says it specialized Kolibri for German, reasoning, math and agentic behavior. What it does not publish is an independent benchmark. The comparisons cited are the company's own internal evaluations, and no third party has confirmed the Pareto claim.

No outside lab has checked any of it yet.

llama.cpp turns decision models into a server endpoint

Also on 3 October, the ggml-org team published a Hugging Face post describing a new /v1/systemone endpoint in llama.cpp. You send a state (text, JSON or a screenshot) plus typed questions, and the model returns a probability for each option in one forward pass. The post says the API follows the System One format introduced with TypeSafe's Jev model, so existing clients need only a new base URL.

The model list is where the on-device angle becomes concrete. Julia-1 answers in 3 ms on one NVIDIA RTX PRO 6000. Laya takes 5 ms. Kev-4B and lev are 4B models at 12 ms and 36 ms. OpenJev is 27B and handles images; Clef is 27B and runs under Apache 2.0. Speeds are medians for a single question on that card, so laptop figures will be slower.

Decision models change what a small model does. Instead of generating text, they score the options you hand them: routing a request, moderating content, checking that an agent's step worked, or picking the next action. No output tokens are generated, which is why the latency figures are in milliseconds rather than seconds.

The community Decision Index shows how they compare.

The post points readers to that index for cross-model comparison, and says more decision models are coming. The endpoint supports three question types: choice, score and noul, a yes/no question that returns the probability of yes. Inputs can also include images for vision-capable models such as OpenJev.

Guardrails get small too

ZeroLeaks released Shield Small on 2 October, a 118M-parameter classifier for prompt injection and jailbreak detection. It runs locally on a CPU with 118 MB of int8 ONNX weights, and inference works offline once the model is downloaded. The model card says to use it on untrusted text before an agent reads it: retrieved documents, tool results, web pages, messages and tool descriptions.

The published numbers come with a caveat the authors state themselves. Shield Small plus rules v2 scores 84.3% mean category-balanced accuracy, 71.8% attack recall and a 7.2% false-positive rate across 35 public datasets. The card says that comparison informed development and is not an unseen evaluation. It also warns the model can miss attacks and flag harmless text, especially security discussions that quote attacks.

There is a second limit worth reading before deployment. Inputs longer than 6,206 tokens get truncated, and the example script scores 28 windows from the start and four from the end, returning coverage.truncated: true. An unflagged, truncated result must not be treated as clearance for the whole document, according to the card. The score is also not a calibrated probability that an attack will succeed.

The consumer end is already shipping

Underdog is the most consumer-facing version of the trend. The company sells a personal assistant that runs on a Mac with Apple silicon, with a Windows beta, and says nothing leaves the machine. Its models are Woof 4B and Underdog 27B. The site claims Underdog 27B beats Claude Opus 4.6 Max on the Artificial Analysis Intelligence Index, a claim that comes from the vendor and cites a third-party index without a published score.

The security paperwork is more checkable. Underdog says it passed all 20 requirements of the App Defense Alliance assessment run by Leviathan Security Group. It says Google approved it to read Gmail and Calendar, and that Hacktron AI finished a penetration test of the Mac app and the website. Jane Manchun Wong is listed as taking the Mac app apart, an audit that is under way rather than complete.

Google, meanwhile, is tightening the free tier of Gemini. A support page updated on 3 October says changes start for users without an AI subscription on 9 October, when Auto becomes the default and most prompts go to Flash-Lite, the fastest model. If smart model selection is off, chats use Flash-Lite for everything. Paid tiers get higher limits: AI Plus at 2x standard, AI Pro at 4x, AI Ultra at 5x or 20x AI Pro.

Read together, the two moves point in opposite directions. Google is pushing free users toward its smallest cloud model and metering them by compute every five hours. Aleph Alpha and the llama.cpp contributors are putting weights in users' hands. Neither replaces the other: Kolibri needs a GPU and an operator, while Flash-Lite needs only a browser.

What is still missing

Independent evaluation is the gap. Aleph Alpha's Pareto claim, Underdog's index claim and Shield's accuracy figures all rest on vendor or self-selected test sets. The llama.cpp post gives latency on one datacenter card, which says little about a laptop. None of these releases ships a neutral third-party audit of quality at the small end.

There is also the question of what small models are for. Aleph Alpha positions Kolibri for sovereign work in regulated industries, ZeroLeaks positions Shield as a filter in front of an agent, and llama.cpp positions decision models as routers and checkers. None of them is trying to be a general chatbot, and the benchmarks that decide whether a chatbot is good do not measure any of these jobs well.

Comments 0

Sources

8
  1. 01Kolibri Has Landed: A Sovereign Open-Weight ModelEN
  2. 02New in Llama.cpp: Decision ModelsEN
  3. 03Shield: A 118M model for detecting prompt injections and jailbreaksEN
  4. 04Underdog private personal on-device local AI for privacy and securityEN
  5. 05Changes to Gemini model access and limitsEN
  6. 06Anthropic's super bug-hunting model Mythos is hardcore good at math, as latest vuln under attack showsEN
  7. 07Anthropic claims popular Chinese AI model has Mythos-class hacking abilitiesEN
  8. 08A Beginner's Guide to Running AI Models LocallyEN

All figures and quotations in this text come from the sources listed below.

Content prepared by the editorial team with AI assistance.

Rachel Nwosu

Rachel Nwosu

AI, models and technology

Rachel Nwosu covers AI, models and technology for FLASH24, working from public model documentation, benchmark releases and repository histories rather than press summaries, and she skips announcements that arrive without reproducible numbers. She checks training-data claims against dataset cards and reruns reported metrics where code is available. She spends much of her week interviewing researchers and engineers, tracking model launch calendars, and comparing vendor benchmarks with independent evaluations. Outside the desk she runs 3D printers, restores old computers, and tests how models learn from internet junk. She does not publish benchmark figures she cannot trace to a source.

Newsroom →

Comments

0
  1. No comments yet — be the first.

Write a comment

Comments are public. We do not publish abuse, spam or advertising.