Search traffic is shrinking and the web is rebuilding around agents

On 29 September, IEEE Spectrum's Computing section published a poem by Ralph Earle, a former IBM senior software engineer, about translating "dry and unrelenting code" into a user interface. It is a small artefact of a large shift: the interface is where the money now moves, and increasingly the interface answers instead of links out.

The numbers behind that shift arrived the same week. A National Bureau of Economic Research working paper issued in September 2026 estimates that from November 2022 to December 2025, AI raised the market's expected present value of software engineering productivity by the equivalent of a permanent 32.6% gain, with a corresponding GDP effect of 3.6% in the baseline and 6.5% when higher software engineering productivity also lifts R&D productivity. The authors, Alex Blumenfeld, Jonathon Hazell, Chen Lian and Andreas Schaab, add that by mid-2026, amid rapid progress in coding agents, the effect had more than doubled relative to the end of 2025. That is a macro figure, not a media one, but it describes the capital flowing into the systems that now sit between users and content.

Full-text search still works, but it no longer finishes the job

Manticore Search published a long post on its blog explaining why keyword search has not broken, and why that is not enough. The company cites Google's own May 2026 disclosure that AI Mode passed one billion monthly users, and argues that queries like "I need black waterproof running shoes for daily runs on wet pavement" contain several tasks: extracting constraints, understanding intent, checking price and availability, and then explaining the difference between options. Manticore's answer is a hybrid: full-text and vector search combined through result ranking, with a language model generating an answer on top. It tested this on ConvApparel, a conversation dataset that after cleanup contained 82,524 products across footwear, pants, tops and outerwear, and built a demo called Manticore Apparel Shop. The post is explicit about the risk: if the system says a model suits rain, the user should be able to open the product and verify the source of that claim. That last sentence is the publisher's problem stated by an infrastructure vendor.

How Meta's shopping agent decides what to show

On 28 September, a researcher writing as Kalan published a reverse engineering study of Muse, Meta's new AI agent, at caeliai.com. Muse launched in early September 2026 and, by Apptopia's count cited in the post, passed 2.8 million installs worldwide in its first 12 days, with about 642,000 daily users in the U.S., faster than ChatGPT's early mobile launch. The study is unusually concrete about the retrieval layer. Kalan reports finding the catalog program at /opt/hatch/bin/meta-catalog-search, a protected worker socket at /run/hatch/privsep/meta-catalog-search.sock, shopping instructions in a file called shopping-SKILL.md, and a ranking_score attached to each product. In the shopping tasks described, Muse returned 40 to 71 products with scores, checked them on retailers' own sites through a browser, and surfaced two or three cards to the user.

Kalan also notes that the raw catalog results included internal debug links pointing at Meta's internal network, and that the one link opened, on 28 September, showed only Meta's internal login page.

I don't have a Meta login, so all I have are the addresses, not what's behind them.

The post is transparent that the decoder was written to read an 11.7 MB compiled Rust program without running it, and that experiments changed one variable at a time and were repeated three times. Treat it as one researcher's account, not a Meta disclosure. Meta's own description, published 8 September 2026 and quoted in the post, says each user's Muse runs in its own isolated section of a dedicated virtual machine with a browser, a working folder, command-line tools, skills and helper agents, and that built-in connectors run in separate protected workers.

The commercial implication is straightforward. If a shopping agent returns 40 to 71 candidates and shows two or three, the ranking function decides who gets seen. Publishers and merchants do not control it.

Search evaluation is still hard, and that matters for everyone downstream

Doug Turnbull's widely read post on LLM-as-judge evaluation, republished on his blog on 28 September, is a useful corrective to anyone assuming these rankings are settled. Testing pairwise relevance on the WANDS furniture dataset, he found that a forced LHS/RHS choice gave 75.08% precision at 100% recall over 1,000 pairs. Allowing the model to answer "Neither" raised precision to 85.38% but cut recall to 17.10%. His fix is simple and worth repeating: ask the same question with the products swapped, and only accept the verdict if the model is consistent both ways. It is the sort of methodological detail that rarely survives a product announcement.

Around the same period, MongoDB announced MongoDB 9.0, Atlas Infinite and Atlas Agent Engine on 29 September, with an interim CEO note on the company blog the same day. The vendor describes Atlas Agent Engine as providing memory, runtime and governance for agents in production, open to any model or framework and built on standards including the Model Context Protocol for tools, A2A for agent-to-agent delegation and OpenTelemetry for traces. Pricing is for public preview and subject to change, the company says.

Two more 29 September posts fill in the plumbing. Unblocked published details of an adaptive router that moves LLM traffic between Baseten, Fireworks and CoreWeave, all serving GLM 5.2. Under round-robin, Fireworks took 51% of tasks and Baseten 49%, even though Fireworks' prices were 25% higher and Baseten was faster, the company writes; under a fixed order, Baseten served 98.5%. CoreWeave's list prices were about 45% lower than Baseten's. And Loro published an argument that CRDTs alone are not enough for local-first sync, describing a case where a conversation index synced but the message body did not, leaving a user staring at a notification with nothing behind it.

Where the readers are supposed to come from

None of this proves that search referrals will collapse. It does show where the investment is going. The NBER paper is a market-implied measure, not a measurement of output, and the authors present it as such. The Manticore and MongoDB posts are vendor arguments. The Muse study is one independent researcher's account with an explicit note that he lacks access to Meta's internal systems.

For publishers, the practical reading is narrower. The answer layer is being built by companies whose incentive is to resolve the query, not to route the reader. If the catalogue behind a shopping agent is scored, filtered and rendered as two or three cards, the traffic that used to arrive from a search results page does not disappear all at once. It gets absorbed one query at a time.