Moving to a CDN? Google explains why Googlebot crawling spikes, and how to keep pages in Search

On November 11, Google published a guide for sites about to start using a content delivery network, or CDN. The headline warning: right after the switch, crawler traffic hitting the origin server can climb almost immediately, which is exactly the problem the CDN was supposed to solve.

According to Google Search Central, the explanation lies in how Google's crawling infrastructure calculates each site's capacity limit, also known as hostload. When responses keep getting slower, the crawlers back off. When responses are fast and stable and there is crawl demand, that limit gradually rises.

Once DNS starts pointing to the CDN, connection and response times drop sharply. To Google's systems, it looks as if the server just got a major hardware upgrade, so the crawlers ramp up their request rate to match.

The cache starts out empty

A faster crawl rate wouldn't be a problem if the CDN absorbed every request. But a new CDN starts with an empty cache. On the first request for each URL, the edge server doesn't have a copy of the file yet (HTML, CSS, JavaScript, images) and has to fetch it from the origin. Only subsequent requests for that same URL are served straight from the edge.

Human visitors cluster around a handful of popular pages, like the homepage, the most-read articles and current promotions, which get cached quickly. Crawlers do the opposite: they work through the site's entire URL inventory, and every address not yet in the cache goes straight to the origin. Google notes that crawlers don't speed up all at once and that many CDNs already smooth out traffic spikes, but until the cache warms up, the origin server is still the one carrying the load.

Seeing that spike in the logs often pushes admins to switch on blanket bot blocking in the CDN dashboard. Without exceptions, those rules treat all automated traffic the same way. When the CDN or the web application firewall (WAF) challenges a request with a CAPTCHA, Google's crawlers fail. The site looks restricted or broken, the crawlers pull back, and pages, including shopping listings, start dropping out of Search.

What Google recommends

• Allow verified bots. Most CDNs and WAFs offer a built-in setting or rule to allow known bots, and Google's crawlers can be identified by the IP ranges the company publishes. Another option is to use robots.txt or rules that target only the bots you don't want.
• Warm up the cache first. Before routing all traffic through the CDN, request through it, at a controlled pace, your highest-traffic pages, pages with expected spikes (promotional or seasonal launches) and your most-crawled pages.
• Use 429 or 503 as a temporary brake. If load on the origin becomes critical while the cache warms up, configure the server or CDN to return 429 (Too Many Requests) or 503 (Service Unavailable) for a short time: a few hours, or 1 to 2 days at most.

For brand teams, the takeaway is that an infrastructure migration also affects your presence in search results. A security rule that's too broad, with no exception for verified bots, can pull pages out of Search and out of other products that rely on crawled data.