Agent browser tooling splits into two camps: what agents can see, and what they're allowed to touch

The newest piece of agent tooling in the dossier is small and unglamorous. agent-desktop, published on 3 November under a Vercel address, reads the accessibility tree of a desktop app and returns it as JSON. Every element an agent can act on gets a reference it can click, type into or wait on. The tool ships as a CLI and a C library for macOS, Windows and Linux, according to its documentation.

The documentation is unusually blunt about scope. "It is not an agent. The observe, act, and verify loop lives in the calling agent or script," it says, adding that there is no model inside and no interactive prompt: every command prints one JSON envelope and exits. It also concedes the limits. The tool "reads accessibility trees and does not match pixels, so it cannot see a UI that publishes no accessibility tree," and it does not automate web pages at all, deferring those to a companion browser tool.

Two surfaces, two sets of problems

That split matters, because the web side of agent automation is now a legal battleground. On Monday, 2 November, the New York Times sued Brave, alleging the browser company scraped its copyrighted content and distributed it to AI companies the suit names as Anthropic, Perplexity, Mistral, xAI and Cohere, TechCrunch reported. The Daily News, Chicago Tribune and Orlando Sentinel filed a similar suit the same day in federal court in Manhattan.

According to TechCrunch's account of the filing, the Times likens Brave to a "supplier, dealer, and distributor of stolen copyrighted content." The suit says the company built an index of more than 30 billion web pages using an unidentified crawler that publishers could not block. It also targets a Brave feature that bypasses paywalls and the company's practice of blocking publishers' ads while selling its own. Brave, founded in 2015, has raised roughly $250 million to date, per private-market trackers cited by TechCrunch. The Times seeks an injunction and damages.

Read together, the two stories describe the same tension from opposite ends. Desktop automation through an accessibility tree stays inside an app the user already runs. Web automation crosses publisher boundaries, and the bill for that is arriving in court.

Who pays for the tokens

Cost is the other constraint, and the dossier puts a number on it. Y Combinator CEO Garry Tan told Walter Isaacson's podcast The Builders, in an episode TechCrunch says was set to publish on Tuesday, that he spends "about maybe $100[,000] to $200,000 a year on tokens." He runs an internal YC agent called Stray Light that holds context on every founder the accelerator has backed. His personal agent, he said, has read his email, Dropbox and Apple Notes going back 20 years.

Tan was careful about the optics. "It's too expensive right now," he said, calling his own spending "a crazy privilege that not everyone can do." That admission sits awkwardly beside vendor claims. Momentic, a YC W24 company that launched on Hacker News on 2 November, advertises 77,813 tests created, 70,607,819 test runs executed and 117,010 bugs caught on its landing page. It sells plain-English YAML tests that a coding agent can author over MCP.

The evidence is thinner than the marketing

Independent measurement is harder to find. MIT Technology Review published survey work on 2 November finding that 48% of insights generated by analytics agents lead to actions that improve the business, up from 37% a year earlier. Lack of business context was cited by 55% of executives as the main reason for poor output. The piece carries a disclosure: it was produced by the publication's custom content arm, not its editorial staff.

Academic work is more cautious still. A paper submitted to arXiv on 30 October by Akshat Jani, Sakhinana Sagar Srinivas and Venkataramana Runkana tested whether agents can build complete executable JSON specifications for geometry creation, meshing, PDE solving and export. On 120 template-related requests the full workflow reached 98.3% correct first-turn specification and decision outcomes, against 92.5% for schema-assisted single shot and 82.5% for model-only, with no false-ready decisions among 42 requests needing clarification or rejection. But only 9 of 12 live user-to-results tests achieved joint specification, execution, export and numerical-reference success, or 75.0%, and the authors flag fresh-response regressions as a consistency problem.

That gap, near-perfect on templates and three in four on live requests, is the number to keep in mind. Tooling like agent-desktop removes some failure modes by refusing to guess: a successful action does not prove the UI changed, so the loop ends with a read.

Alignment research is moving in a similar direction. MIT Technology Review reported on 3 November that researchers are pursuing "reverse game theory," or mechanism design, to shape agent incentives rather than inspect model internals, citing a preprint by MIT economist Stephen Morris and colleagues at Yale and Columbia on cross-checking agents to stop sandbagging. Morris told the publication that "AI agents have a very good understanding of what other AI agents can achieve."

None of this settles whether agents can be trusted with a browser. It does suggest the industry has stopped asking whether they can click, and started asking what they are allowed to see.