Trillium Labs opens up risky AI research as OpenAI's safety exits pile up
Two industry scientists have launched Trillium Labs, a nonprofit that will publish details of high-risk AI experiments, including recursive self-improvement and agents, so outside researchers can replicate them, WIRED reported.

The launch lands in an awkward week for the labs Trillium is implicitly arguing against. On Saturday, David Robinson, a leader on OpenAI's Safety Systems team, published an essay in The Atlantic. He said the company is not being careful enough. Business Insider, which first reported his resignation on Friday, said he quit the previous week.
Trillium's founders, Nathan Lambert and Tom Zick, want the opposite of that arrangement: research that outsiders can check. The pitch is that secrecy has a cost. If only a handful of people can see how a frontier model was built and tuned, the argument goes, nobody else can test whether the safety claims hold. That is the whole case in one sentence, and it is also the part that is hardest to prove. A lab that publishes its methods invites rivals to copy them. A lab that stays quiet invites nobody to check anything. Trillium is betting the second cost is larger.
What changes for readers is the yardstick. Buyers of AI products, and the regulators circling them, currently judge safety mostly from system cards and company statements. Trillium promises experiments that outside scientists can study and replicate. That turns a marketing claim into something closer to a testable one. What is still unknown is how much of its own research the nonprofit will publish, and who pays for it. Neither founder has said how the organisation is funded.
Lambert told WIRED that closed development is the wrong direction. "Over the past few millennia, humanity has had the scientific method in our toolbox as a way to mitigate harms and build better futures," he said, according to WIRED's report. "The current closed trajectory of frontier AI development is taking us a step backwards."
Open weights are already winning users
The open approach is no longer a fringe position, because open models have taken a large share of real usage. Chinese open-weight models grew from under 2% of global usage in late 2024 to more than 45% by June this year, according to OpenRouter data cited by CBS News. By August, Alibaba's Qwen had passed three billion downloads worldwide. Those two numbers sit awkwardly together: a jump of more than forty points in global usage in under two years, and a single model family passing three billion downloads within months of that survey. Whatever the safety debate concludes, a very large number of people already run weights they can download and inspect, or misuse, without asking anyone.
That scale matters for the safety argument. CBS News reported that Hirundo, an Israeli cybersecurity startup, found Qwen will not discuss the 2019 Hong Kong protests or Tiananmen Square in 1989. The model calls Falun Gong a "dangerous cult" and deflects questions about Winnie the Pooh. When CBS asked the model about forced labour camps for Uyghurs, it said none exist. Alibaba did not respond to a request for comment, the network said. Uber Eats has said its search and delivery functions are built "on a Qwen backbone". Airbnb's Brian Chesky told the LA Times his company relies heavily on the model for customer service.
Put plainly: downloading a model does not make its behaviour transparent. Trillium's bet is that published methods let researchers catch that kind of baked-in bias, or at least argue about it in public.
The founders have form. Lambert worked at the Allen Institute for AI and at Hugging Face, runs a technical blog and started American Truly Open Models, a push for US companies to release more open models. Zick worked at Harvard and helped Charles Schwab write responsible AI policy. They met on Zoom during the pandemic as graduate students at UC Berkeley.
Trillium will start with post-training, the fine-tuning that happens after a model is built, and with recursive self-improvement, where AI contributes to its own research. Both are areas where labs tend to keep quiet. WIRED noted that Xiaomi has published live details of a major training run, and that Stanford researchers are pretraining a model called Marin in the open.
Inside OpenAI, the exits keep coming
Robinson's departure is the freshest data point in the argument over who gets to check the work. He helped develop the system cards OpenAI publishes with its models. In his Atlantic essay he wrote that he agrees with other recently departed staff. The companies building the technology "aren't being nearly careful enough", he wrote. As OpenAI "sprints from one launch to the next, it is failing to achieve the level of care that I believe is needed", he added.
OpenAI pushed back in a statement. It said it is "making sure our models don't become more capable than we can safely manage and secure, and we pause training or hold back models when we need to slow down." The company said it is expanding work with outside evaluators. It is also improving real-time monitoring to catch concerning behaviour earlier in training.
"An environment where things like this can happen is no place to grow artificial minds that could be smarter than we are and that might not do what we want them to."
That line, from Robinson's essay, follows a series of safety incidents OpenAI has acknowledged. The company also said on Thursday that it had "parted ways" with three researchers over violations of its policy on sharing sensitive information. Johannes Heidecke, its safety head, left earlier this year. At OpenAI's developer conference on Tuesday, CEO Sam Altman said safety and alignment need to stay ahead of model capabilities. Read the sequence in order and the pattern is hard to miss. A safety lead gone, three researchers out over information sharing, a system card veteran resigning, all inside a single year. The company's public answer is more outside evaluators and better monitoring.
There is a counter-argument, and it is not fringe. Critics of slowing down warn that if Silicon Valley pumps the brakes, international rivals and bad actors gain ground. That tension sits underneath every evaluation proposal. The people best placed to audit a model are often the people with the least access to it.
Evaluations are getting harder to trust
Academic work points at the same gap from another angle. A benchmark called ScholarCatalyst was built from labels by 184 lead authors of 207 recent computer science papers. It tested whether AI systems can find the earlier paper that shaped a project. Agentic search scored no better than plain embedding retrieval, 0.42 against 0.48 Recall@20, even though the agent called that same retriever as a tool. An agent built on Claude Fable 5.1, which may have seen the finished papers during training, reached only 0.51.
The authors describe the task as a step toward scientific agents that can take a half-formed idea and point to the research it needs. The result is also a warning about evaluation design: models can look capable on a task when what they are doing is recall.
Meta's research arm made a related admission. It published six papers produced with mathematicians using its Muse Spark models in Thinking Mode through the ordinary meta.ai chat interface, five of them answering previously open questions. Meta said that after finishing the work it learned other teams had independently announced solutions to some of the same problems. It said the papers mark which passages were drafted by researchers and which by AI. One proof strategy, for a Gaussian ellipsoid fitting threshold, was developed with the model under a mathematician's guidance and checked by four others.
Publishing the method, the credit and the concurrent work is exactly the norm Trillium says it wants to scale. Whether a nonprofit can do that for agents and self-improvement, where the risks are less tidy than a geometry proof, is the open question.
The academic mood is not helping. A recent thread on the TYPES mailing list, summarised by one programming languages researcher on 2 October, described morale in the field falling sharply in autumn 2026, with stress comparable to the pandemic. The same researcher argued the tools make bigger problems tractable. Both readings point the same way: the work of checking AI is moving outside the labs, whether the labs like it or not.
Sources
8- 01These AI Experts Want to Do High-Stakes Research Out in the OpenEN
- 02OpenAI safety leader David Robinson resignsEN
- 03Some topics are off limits inside popular Chinese-made free AI, researchers findEN
- 04Solving Open Research Problems TogetherEN
- 05ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New ResearchEN
- 06PL research is dead, the age of PL exploration is just beginningEN
- 07How to apply to AI safety fellowships (and beyond)EN
- 08Memory safety for Postgres extensions in C/C++EN
All figures and quotations in this text come from the sources listed below.
Content prepared by the editorial team with AI assistance.
Comments
0- No comments yet — be the first.