Microsoft and Hugging Face Put Agent Reliability at 79,853 Failed Checks
Microsoft and Hugging Face published ThinkingBox on 3 October. The benchmark grades AI agents on the database records they leave behind, not the answers they write. Across 507 business workflows, 79,853 of 121,680 valid trials failed its executable checks.

Microsoft and Hugging Face built the benchmark together, as set out in a Hugging Face blog post dated 3 October. Each of the 507 stateful tasks runs 20 times against a clean backend, across 12 LLM models. The premise is plain: a finished reply is a claim, and the database is the evidence.
Why it matters depends on where you sit. Teams running agents against billing, ticketing or inventory systems get an argument from the post. Final responses and valid tool calls are only proxies, it says, and the end state is what should be tested. Buyers of agent platforms should expect vendors to field questions about repeatability, not just demo quality. Neither the post nor the authors say which of the 12 models failed most often, or how the results map onto any commercial product.
The headline number is stark. A common-set ablation covered 121,680 valid trials across 12 LLM models. Of those, 79,853 attempts failed the executable checks.
Clean endings, wrong records
Of those failures, 67.24% still terminated cleanly, invoked a state-changing tool and reported no final tool error, according to the Microsoft and Hugging Face write-up. Executable checks found wrong field values in 77.61% of them, unintended extra effects in 43.30% and missing required effects in 25.36%, and those findings overlap. The authors walk through a retail case adapted from a benchmark task, sandbox_external_retail_group1.py:test_case_ST003_006. An agent handles a customer's stuck $745 kitchen appliance in nine tool calls and reads the refund policy correctly. It then closes the ticket as resolved when the required end state was on hold. One field check fails.
The benchmark ships for others to run. It goes through OpenEnv, and the post says readers can execute the case themselves.
ThinkingBox is not the only recent attempt to pull agent evaluation away from chat quality. A practitioner write-up from Sarthak Gaud lays out a four step loop for evals. Freeze a snapshot of every agent run, have a subject matter expert mark each step correct or incorrect, compile the annotations into a dataset, and treat code checks as the first line before bringing in a judge. Gaud suggests starting with around 20 runs. That is a small number next to 507 workflows run 20 times each, but the principle matches: store what happened, then check it.
Oversight has a limit too
The other half of the problem is the human in the loop. A paper by Emre Turan, posted to arXiv on 8 June, argues that approval gates are the easy part. Judging which actions to stop is the hard one. On a hand-labeled set of 125 adversarially weighted agent actions, reviewers only moderately agreed on what counted as risky, with a Fleiss' kappa of 0.52. Model the reviewer as fatiguing under escalation load, and realized safety follows an inverted U in the escalation rate: more human oversight can make a system less safe. The paper calls the inverted U and a flooding attack modeling results that motivate a human study.
"A trajectory is a claim. Database state is the evidence. Repetition is the trust test."
That line, from the Microsoft and Hugging Face post, sums up the shift. Repeatability is the test now, and a single good run proves little.
Some teams are already reorganising around that assumption. Alexey Indeev wrote on 4 October that he merged 562 pull requests in two weeks without reviewing every line. He leaned instead on tests, CI and review bots that an agent cannot bypass. His own warning: AI code is non-deterministic, and so is human code.
Others point at a different moving part. A blog post on agent drift published 3 October asks why an agent's behaviour changes when nobody touched the prompt, the code or the model. Its answer is that models, harnesses, tools and data all sit in a system of systems, and any of them can move. The post cites NIST guidance on post-deployment monitoring for generative AI, plus Microsoft lifecycle guidance that treats foundation models as versioned dependencies.
Tooling vendors are answering at different layers. Apple said in a developer blog post reported by 9to5Mac on 2 October that it will add controls so that granting Full Disk Access requires very explicit user action. AgentMail, in a post on 4 October, argues agents should get their own inbox and sign-in key rather than borrowing a person's password. That way a misbehaving agent can be cut off without locking out its owner. Kimchi, launched on 4 October, sells an enforcement layer that halts agent spending at per-user, per-team or per-key limits.
The common thread is measurement. ThinkingBox reports three numbers per task, and the post does not disclose how the 12 models were selected or what pass rates look like per model.
For anyone putting agents into production records, the practical read is blunt. Check the database, not the transcript, and run the task more than once.
Sources
8- 01The Agent Said It Was Done. The Database Disagreed.EN
- 02Writing Evals for Agentic Systems as a 4 Step LoopEN
- 03Oversight Has Capacity: Calibrating Agent Guards to a Subjective Fatiguing HumanEN
- 04I stopped reviewing my agents' code. Here's what I do insteadEN
- 05AI Risk - A user's guide (Part IV): Understanding Agent DriftEN
- 06Apple says it's tightening macOS privacy controls amid the rise of AI agentsEN
- 07Give Your AI Agent Its Own IdentityEN
- 08Pi-based coding agent with hard budget capsEN
All figures and quotations in this text come from the sources listed below.
Content prepared by the editorial team with AI assistance.
Comments
0- No comments yet — be the first.