Google data: a new page is discovered in about 20 hours, refreshed every 30 days

Gary Illyes of Google showed internal data on how long crawling, indexing and serving actually take. He presented it at a three day Search Central Live Deep Dive event in Europe, according to a write-up by Barry Schwartz at the Search Engine Roundtable on 5 October. The numbers were split into a fastest time, a typical time and a slowest time.

The headline figure: a new URL is typically discovered in about 20 hours. The slowest case runs to weeks, or never.

That gap is the part publishers should read twice. Discovery is not indexing, and indexing is not ranking or traffic. Illyes warned that many of these processes are linked, so delays stack up. A page cannot be indexed until it has been crawled. It cannot earn a click until it has been indexed and ranked. Schwartz also quotes French SEO Neil McCarthy, who posted more on LinkedIn, and John Campbell of We Are Roast. Campbell called the data "brand new information" in his notes from the event.

What the timings say, line by line

The table Schwartz published covers far more than crawling. Refreshing a known URL typically takes about 30 days, with a slowest case of weeks to never. Sitemap processing is usually about 24 hours but can stretch to 14 days, or never on quality grounds. A robots.txt update lands in about 24 hours, with 25 hours as the slowest case.

Crawl capacity updates take 4 hours to 1 to 2 weeks typically, and 1 to 3 weeks when Google is in recovery. Crawl demand updates run about 20 hours typically, weeks to months at the extreme. Crawl capacity itself can drop in seconds when Google backs off, for example if a server struggles. That is a warning for anyone whose infrastructure wobbles under load.

Indexing end to end is typically about 1.5 hours, but the slowest case is months or never on quality grounds. Canonicalisation changes take 1 to 3 weeks typically and months with conflicting signals. A site move runs 1 to 3 months typically and 6 months to 1 year or more at the extreme. Removal from the index takes 1 to 3 weeks typically, months at the slowest.

On the serving side, title and snippet updates typically take 1 to 2 days, with several weeks to months as the slowest case. Meta annotations take 45 to 90 minutes typically and 1 to 4 days at the extreme. Link annotations take minutes to 1 to 3 weeks typically and months at the slowest. Core update recovery runs 3 to 6 months typically, with 6 months to 1 year as the slowest case, namely the next core update.

Illyes was careful about how the slides should be read. "Mind that this was an exercise to see if the audience can relate to the numbers we pulled internally and put in those slides," he told Schwartz after the post went up. So these are illustrative internal figures, not a service level agreement.

Why publishers care about crawl timings

For a news site, the typical numbers matter less than the slowest ones. A story that is discovered in 20 hours is already dead on social. A refresh cycle measured in about 30 days means a corrected headline may not be reflected for a month. An updated developing story can wait just as long. A canonicalisation change measured in weeks can keep duplicate versions of a story competing with each other for the same query.

The serving figures cut the other way for anyone chasing clicks. If a title update takes 1 to 2 days typically, then tweaking headlines for search is a slow lever, not a fast one. The same goes for structured data updates, which the table puts at hours to 1 to 2 weeks typically and weeks or never in the slowest case.

The bigger question: where the traffic went

Crawl timings are a technical story. The commercial story is that publishers are already arguing about a fall in search referrals. Google is now handing them a document that explains the plumbing without addressing the outflow.

That argument has moved into court. Recent headlines note that a judge dismissed cases brought by Chegg and Penske over AI Overviews, with the reasoning summarised as an expectation not being an agreement. They also note that Google won dismissal of lawsuits about AI search's impact on website traffic. Those headlines are context from the news wire, not facts established in this article. The underlying rulings should be read directly before anyone cites them as precedent.

What the figures do establish is that the measurement industry around search is being rebuilt. Search Engine Land reported on 5 October that AI usage has surged sixfold for local search, fragmenting the digital ecosystem. It separately covered OpenAI bringing visual ads and expanded measurement to ChatGPT. Digital Information World reported that Google AI Overviews appeared in 13.7% of searches, with 11% unsupported claims and potential publisher revenue risks.

None of those numbers come from Google. Treat them as third-party estimates with their own methods. Note that they disagree in emphasis with Google's own framing, which presents AI features as an addition to search rather than a subtraction from referrals.

What a publisher can actually do with this

The practical reading of Google's table is that the cheap wins are on the discovery side. Sitemaps, robots.txt and crawl demand all resolve in about a day in the typical case. A publisher that fixes those first sees movement fastest. Canonical and site move work is a quarter scale project, and should be planned as one.

Second, the quality caveat repeats across the table. Sitemap processing, indexing, structured data and video analysis can all end at never on quality grounds. That is Google's polite way of saying that a page it does not want to index will not be indexed, no matter how fast the crawl is.

Third, nothing in the table promises traffic. It describes how content gets into the index and how changes propagate. Whether a page is surfaced, and whether a user clicks it rather than reading an AI summary, is a separate system. That system has separate rules that Google has not published in comparable detail.

The wider technical context is a reminder of how much of the web's discovery layer is now machine-mediated. Cloudflare launched a Web Search API in beta on 2 October, letting AI agents and applications search the internet through providers Ceramic.ai, Exa and Linkup. Search requests are logged and billed through its AI Gateway, according to the company's developer documentation. Every provider returns a link to the source of each result, Cloudflare says.

That is a different pipeline from Googlebot, and it does not depend on crawl budget or a 30 day refresh. It is also, for now, a beta with three providers.

For publishers, the two systems point in opposite directions. Google's data says the classic index is slow to update and can decline to index at all. The agent search layer says answers can be grounded in live results with a single API call. Which one carries the readers, and the advertising, is the question the crawl table cannot answer.