Postgres Search Engine Anatomy Lands as Publishers Rebuild for Post-Search Traffic

The post landed at 01:39 UTC on 27 September. PlanetScale's engineering blog published "Anatomy of a (Postgres) Search Engine," a walkthrough of how inverted indexes, term dictionaries and postings lists replace the row-by-row scans that make LIKE '%postgres%' queries expensive. The timing is not incidental to the media business, even if the post never mentions publishing.

Search traffic is the thing publishers are trying to replace or rebuild, and the infrastructure conversation has moved down a layer. PlanetScale's piece is a reminder that the machinery behind search is not magic. A b-tree index sorts a column's values in a total order and can find an exact value, a range or a prefix. It cannot match content in the middle of a string. That job falls to an inverted index, which maps each individual word in a text field to the documents that contain it.

The post is detailed enough to be useful to engineers and specific enough to be checkable. PlanetScale describes a term dictionary that maps every term across all documents to postings lists, with the dictionary stored on disk as a b-tree or another lexicographically sorted structure. Wildcard and fuzzy queries may scan part or all of that dictionary: g* looks up both gonna and give, while u~2 looks up every term within two character edits, which in the post's example means you and up.

Why the plumbing matters to publishers

Postings lists are almost always sorted and compressed. PlanetScale gives a worked example: an index with 300,000 documents, of which 100,000 contain the word "who," would take 19 bits per posting if each ID were stored literally, but the average gap between successive sorted IDs is three, which fits in two bits. Dense lists may be stored as bitmaps instead, one bit per document. Union and intersection of sorted lists run in a single O(n) pass, and bitmap operations can use vector instructions to OR or AND 128, 256 or even 512 bits at once.

That is the same class of problem publishers now face on the consumption side. On 28 September, Digiday published a recap of its Publishing Summit describing how publishers are rebuilding for the post-search era. The same morning, cxotoday.com ran a piece arguing independent media is being throttled by Google and other AI giants. Neither outlet is named in the PlanetScale post, and the technical blog does not address referral traffic at all. The connection is structural: search is being rebuilt on both sides of the query.

The numbers circulating in the media trade press are not gentle. Recent headlines tracked in the dossier include a 40% year-on-year decline in Google Search referrals for publishers, dated 24 September, and a separate report of search engine traffic falling across the web with some small websites seeing a 60% drop. Those figures come from headline aggregation rather than the primary sources, so treat them as directional. The direction is consistent.

The open-source alternatives are moving too

Luxir, an open-source hybrid search engine published on 22 September, pitches itself on the same economics. Its site claims full-text, vector and faceted search with analytics from one engine, with a work-stealing scheduler that shares indexing, merging and query execution across all cores, asynchronous network IO, and SIMD paths for postings decoding, scoring and vector distance. Facet counts are exact by default, according to the project, and top-k requests prune with block-max bounds when the caller does not ask for an exhaustive total.

The Luxir page is a product site, not a benchmark paper, and its performance claims are self-reported. The same is true of the PlanetScale post, which is published by a company selling a Postgres-compatible database. Both are useful as descriptions of architecture and as statements of intent. Neither should be read as an independent measurement.

Elsewhere in the dossier, a 22 September arXiv paper from Jochen Madler at Sitefire, "SlopShape: Identifying AI-Generated Commercial Web Content," offers a structural detector for AI-written marketing copy. The paper reports 2,250 pre-ChatGPT human blog posts from 268 company domains against 11,250 AI mirrors from five frontier models, detected at 98.0 macro-F1 on held-out companies from 187 structural features alone, with attribution to the correct source model at 79.3% against a 16.7% chance rate. Human-human kappa was 0.928 and human-model 0.946. The paper is a v2, revised on 17 September.

That matters for the same reason the search infrastructure does. If a large share of commercial web content is machine-generated in a tidy, self-announcing shape, then the corpus that search engines index is changing at the same time the engines are. Publishers rebuilding for a post-search era are not just competing for placement in a results page. They are competing for placement in a corpus that increasingly writes back.

Two smaller items in the dossier point the same way. ParkourNote, a solo-built research workspace posted on 28 September, imports papers, books, web pages and videos, transcribes and translates them, and exposes the library to Claude, ChatGPT or Codex through an MCP server. GPU-font, also posted on 28 September, runs a small neural network in the browser via WebGPU to match fonts from an image. Both are individual projects, not platform shifts. They are also both examples of the same pattern: more capability, less server, fewer intermediaries.

For publishers, the practical question is unchanged. Search referrals are falling, the technical cost of building a search product is falling with them, and the volume of synthetic content competing for the same index slots is rising. PlanetScale's post explains how the index finds a word. It does not explain who gets found.