Microsoft's ThinkingBox finds agents leave wrong records even when they report success
Microsoft and Hugging Face published ThinkingBox on 3 October. The benchmark grades AI agents on the database records they leave behind. Across 121,680 valid trials and 12 models, 79,853 attempts failed its executable checks.

The benchmark runs 507 stateful business workflows, each one 20 times from an identical clean backend. It ignores what the agent says and checks what the agent wrote: field values, required effects and side effects. Microsoft published the paper jointly with Hugging Face. The blog post carrying the results is dated 3 October.
The failure numbers matter more than the pass rate. Of the 79,853 failed attempts, 67.24% still ended cleanly, called a tool that changes state, and reported no final tool error. Executable checks then found wrong field values in 77.61% of those cases. Unintended extra effects showed up in 43.30% and missing required effects in 25.36%, with the categories overlapping. In plain terms: an agent that says it is done is not evidence that anything was done.
The worked example in the paper is a customer whose $745 kitchen appliance sat in a courier exception at a Nashville distribution center, fifteen days past its estimated delivery date.
The agent made nine tool calls and read the refund policy twice. It checked the account segment and correctly found it did not qualify for late-delivery compensation. Then it closed the ticket as resolved. The required end state was a hold, pending resolution. That is the gap the benchmark exists to measure. A grader watching tool calls sees nine well-formed calls. A grader checking whether the agent wrote to the database sees that too. Only the record disagrees, and the record is what the customer experiences.
Why it matters for anyone buying agent capacity
Enterprise buyers now price agents on tasks completed. ThinkingBox argues the useful unit is a repeatable end state. Every task in the benchmark runs 20 times from a clean backend. An agent that processes a refund correctly once and mangles it on the next four runs is not scored as a working refund agent.
What is still unknown is how these rates move with harness design, tool schemas and model choice. The headline figures aggregate 12 models without a per-model breakdown in the published summary.
The measurement problem is drawing attention from several directions at once. On 8 June, Emre Turan posted a paper to arXiv arguing that human approval gates are the easy part of agent safety. On a hand-labeled set of 125 adversarially weighted agent actions, reviewers agreed only moderately on what counts as risky, with a Fleiss' kappa of 0.52. Turan models the reviewer as fatiguing as escalation load grows. He finds safety becomes an inverted-U in the escalation rate: more human oversight can make a system less safe. His paper claims none of the mechanisms as novel and cites prior work including FALCON and DeCCaF.
Two different critiques land on the same weak point.
ThinkingBox says final responses and valid tool calls are proxies. Turan says the human reviewer is not an infinitely available oracle. Both point at the same missing piece: a check that holds when the agent and its supervisor are both fallible.
The tooling rush around the benchmark
Vendors are already selling the layer ThinkingBox tests. Kimchi, a coding agent platform built on Pi, advertises a single enforcement layer for spending, data, deployment and model routing, with hard caps by user, team, API key or model. Its own marketing shows a team budget at $4,820 of $5,000 and a user at $820 of an $800 cap with a hard stop.
Spawned takes a different route. It provisions infrastructure into a customer's own AWS, GCP or Azure account, with resources exportable and Terraform-compatible. It lists Google Cloud and Kubernetes as available, and Azure as closed beta. Neither company has published independent results on whether those controls reduce the failure modes ThinkingBox measures. Budget caps stop a runaway agent spending money. They do not stop it writing the wrong ticket status.
An agent can sound correct while leaving the wrong value, changing the wrong record, or creating an extra side effect.
Apple moved on the access side. In a post on its Developer website, Apple said Full Disk Access on macOS largely sidesteps its privacy controls. It said some developers use it in ways that could put users at risk, exposing files, mail, messages and browsing history. Apple said it will add controls so users grant that access only through very explicit action. It has not said what those controls are or when they ship.
The common thread across the benchmark, the arXiv paper and Apple's note is that agent output has to be checked against something outside the agent. ThinkingBox puts the check in the backend state. Turan puts it in a measured escalation curve. Apple puts it in the permission prompt. None of them closes the gap on its own.
Sources
10- 01The Agent Said It Was Done. The Database DisagreedEN
- 02Oversight Has Capacity: Calibrating Agent Guards to a Subjective Fatiguing HumanEN
- 03Pi-based coding agent with hard budget capsEN
- 04Spawned: AIO cloud platform for humans and agentsEN
- 05Apple says it's tightening macOS privacy controls amid the rise of AI agentsEN
- 06AI Risk - A user's guide (Part IV): Understanding Agent DriftEN
- 07Writing Evals for Agentic Systems as a 4 Step LoopEN
- 08I stopped reviewing my agents' code. Here's what I do insteadEN
- 09Let AI agents use your site with 3 HTML attributesEN
- 10Give Your AI Agent Its Own IdentityEN
All figures and quotations in this text come from the sources listed below.
Content prepared by the editorial team with AI assistance.
Comments
0- No comments yet — be the first.