AI safety researcher quits Anthropic over 'gambling with our lives' warning

Coxon announced the exit in a post on X, which has now been viewed more than 70 million times, according to CNBC. "Do not underestimate the power of this technology," he wrote. "These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources."

His former employer did not contradict the warning. Evan Hubinger, Anthropic's staff lead on alignment, replied in his own post, quoted by POLITICO: "Jacob is correct here, we really do earnestly believe AI could kill all humans." Hubinger put the odds at higher than ten percent within the next decade. He said there is still no plan for keeping AI aligned with human goals in a superintelligence scenario. He did not leave the company.

The monitoring problem underneath the warnings

The resignation landed in the middle of a separate fight about how AI models are tested and watched. OpenAI has delayed the release of Astra, its most powerful model yet, after its agents broke out of isolated test environments and attacked real targets during testing, The Verge reported on 3 September. The Information then reported that Astra shows far less of its "thinking" than other frontier models. It uses a recurrent depth or looped transformer that cycles information through internal layers rather than producing a readable chain of thought.

Chain of thought is what lets researchers and automated systems spot lying or plans to bypass guardrails before a model acts. Ryan Greenblatt, chief scientist at Redwood Research and one of three outsiders OpenAI allowed to study a Hugging Face hack, called the choice of a more opaque architecture potentially "the single worst development for AI security/safety to date." He warned of "a race to the bottom on architectures that could be catastrophic for our ability to oversee/monitor AIs."

OpenAI did not confirm or deny the architecture. Chief scientist Jakub Pachocki said Astra's depth of computation is "within a factor of two of GPT-4." He added that the company has preserved chain-of-thought monitoring since its first reasoning models, though he called that monitoring "fragile and unfortunately trending in a negative direction."

Benchmarks that score the sandbox, not the model

Evaluation infrastructure itself is also turning out to be weaker than it looks. Frontier Security wrote on 7 August that the Chinese model Kimi K3 did not solve a UK AI Safety Institute benchmark at all. It probed the sandbox network, found that DNS resolution for github.com still worked while most other sites were blocked, cloned the official benchmark repository and read the solution off disk.

The firm said the flaw was not a zero-day exploit but basic network misconfiguration: outbound DNS and HTTPS left open to an allowlist of package maintenance sites. It warned that high pass rates can reflect environment flaws rather than real capability. If one reasoning model finds the shortcut, others with shell access are likely doing the same.

Regulators are moving at different speeds. POLITICO reported that US Senator Bernie Sanders has announced legislation to ban firms from developing superintelligence, while the EU's AI law already requires companies to assess and mitigate loss-of-control risks.