Cloudflare and Googlebot: blocking AI crawlers with care
Translated from the Spanish original. Leer en español
Blocking training crawlers on Cloudflare can also slow down Googlebot. What changes on September 15, 2026 and what to check on your CDN.
In brief
- Cloudflare splits automated traffic into Search, Agent and Training. If a crawler falls into several categories, the most restrictive rule applies; Googlebot, Applebot and BingBot are the examples the company itself gives.
- Marking Training as blocked can drag Googlebot along, unless you add the corresponding exception.
- Search stays allowed by default, so saying Googlebot will be blocked everywhere is an exaggeration.
- Who lets a crawler through is no longer decided only in robots.txt, meta robots or Search Console: the CDN/WAF layer acts before the request reaches the origin.
The change: who suffers it and who doesn't
This edition was written on September 14, 2026, one day before Cloudflare changed its controls for AI crawlers. The comfortable headline ("Cloudflare blocks AI by default") misleads; what affects SEO is in the fine print about mixed-use crawlers.
On July 1, 2026, Cloudflare announced that from September 15 new domains on its platform would start with Training and Agent blocked by default, always on pages with advertising. Search stays allowed. The same applies to Free customers who already existed and had never touched those settings.
Seen this way, the effect on SEO is small and coherent: crawling that discovers is separated from crawling that trains. The knot is in crawlers with several uses. When one takes part in more than one category, Cloudflare applies the most restrictive rule and mentions Googlebot, Applebot and BingBot as examples of bots that mix Search with Training. As a consequence, a site that blocks Training can end up blocking Googlebot. Nobody deduced it from the outside: it is written in Cloudflare's documentation.
Search, Agent and Training: three uses, sometimes a single bot
The classic mental model was simple: Googlebot requests a URL, the server answers, robots.txt delimits what can be crawled and indexing directives govern what shows up in Search. With AI, a bot no longer has to pursue a single goal: it can index, it can collect to train or it can come in real time on behalf of a user. Cloudflare splits traffic according to what each bot does:
| Category | What it does |
|---|---|
| Search | Collects or indexes content to answer questions later. |
| Agent | Automated systems that act in real time on behalf of a person. |
| Training | Collects content to train or fine-tune models. |
If you cannot see all the columns, scroll the table sideways.
The problems start when a single bot appears in two rows at once.
What is documented
Six points rest on official Cloudflare and Google documentation:
- Cloudflare's new AI Crawl Control controls split automated traffic into Search, Agent and Training.
- Search is not blocked by default. What is restricted, in certain configurations and pages, is Training and Agent.
- Cloudflare acknowledges that Googlebot, Applebot and BingBot can be treated as multipurpose, and the strictest rule applies to those bots.
- Whoever chooses to block Training also blocks the multipurpose bots in that category, Googlebot included, unless they configure the exception.
- Google offers Google-Extended as a separate control: it limits certain uses of content in Gemini and Vertex AI without affecting inclusion in Google Search and without counting as a ranking signal.
- AI Overviews and AI Mode still hang on Googlebot: according to Google, the page has to be indexed and eligible in Search, and it is advisable to allow crawling both in robots.txt and at the CDN or hosting.
The level where it is decided: network versus robots.txt
Google-Extended expresses a preference about the use of content. It is not an HTTP crawler with its own identity: Google clarifies that it has no request user-agent, so crawling is still done with the usual user-agents and Google-Extended works as a control token. That way, Googlebot can continue with Search while declaring that certain uses tied to Gemini are not authorized.
Cloudflare works one layer below. If it cuts a Googlebot request at the edge because it also considers it Training, that request dies before the origin: no document, no robots.txt to read afterwards, no HTML to render. The diagnosis has to start earlier.
We have spent years entering the chain too late, with our eyes on robots.txt, canonicals or noindex. With so many anti-bot controls, that is no longer enough.
Between the headline and the fine print
Some SEO analyses talk about a "trap" for Googlebot or sites that could disappear from Google. The substance is right, but two things need nuance.
- "Cloudflare will block Googlebot by default." Not as stated. What happens depends on the domain, the account type, whether someone already changed the settings and, for the new defaults, on the pages showing advertising. A paid site that already existed and didn't block Training shouldn't assume it will stop being crawled on September 15.
- "Blocking AI equals disappearing from Google." Neither. Everything depends on what is blocked: you can set a position on training with Google-Extended and leave Googlebot's normal crawling intact.
My reading is that infrastructure rules created to manage AI can have collateral damage in SEO when the provider puts several uses under a single crawler.
How to apply it: crawler access audit
This is how I would order the work, with what to look at in each step:
| Step | What to review |
|---|---|
| 1. Real settings in Cloudflare | AI Crawl Control; the three blocks Search, Agent and Training; the old "Block AI bots"; exceptions for multipurpose bots; custom WAF or Bot Management rules. |
| 2. Check Google's access | Crawl Stats in Search Console; URL Inspection on key pages; server logs; Cloudflare Security Events; 403 or 429 codes on verified crawlers. |
| 3. Confirm the identity | Anyone can fake a user-agent. To investigate blocks or set up exceptions, use Google's official verification: published IP ranges or reverse DNS. |
| 4. Decouple Search from Training | If you want organic visibility with limits on training, cutting Googlebot at the edge is a disproportionate measure; that is what Google-Extended is for. |
| 5. Put it in writing in SEO | A security tweak made by IT can turn, weeks later, into an organic drop blamed on content, indexing or the algorithm. Classic corporate sport: each team breaks the other's work. |
If you cannot see all the columns, scroll the table sideways.
Watch out for an uncomfortable detail: if Cloudflare stops the request before the origin, the server logs can be empty. Googlebot not showing up is information too.
This was my work sequence at the time, with September 15 as the reference date:
- List the domains that go through Cloudflare.
- Open AI Crawl Control before September 15.
- Make sure Search is allowed.
- Look at the Training blocks and how they treat multipurpose bots.
- Check whether the old "Block AI bots" is still active.
- Save a baseline: Crawl Stats and, if you have them, Googlebot logs for the previous seven days.
- Measure again between September 16 and 18 and compare.
If there are already symptoms of indexing loss, the diagnostic route is in if Google started to deindex your pages, where would you look.
Car rental and travel: the sector where it weighs most
Here the relationship is direct because comparison sites and travel websites combine three things: many transactional URLs, dependence on frequent crawling and fairly aggressive anti-bot protection at the CDN.
In car rental, moreover, destinations depend on inventory, prices, airports, offices and geographic combinations that move often. If Googlebot crawls much less, the loss of rankings can come late; other warnings appear first:
- pages that take longer to refresh;
- canonicals that are reprocessed more slowly;
- new routes or destinations that are discovered more slowly;
- changes in titles, content or internal linking that take longer to show;
- difficulties in huge programmatic architectures, where Google must crawl millions of combinations.
For a large comparison site I would set up a crawl health alert that asks more than whether Google indexes:
- how many verified Googlebot requests arrive each day?
- with which response codes?
- what share reaches categories, destinations and transactional pages?
- did that pattern move after touching the CDN, WAF or bot management?
On large sites, discovering on day one that crawling drops is much cheaper than finding out three weeks later from a traffic drop.
How I see it
More than a one-off piece of news, this is an architecture warning. SEO teams have looked at anti-bot infrastructure as a systems or security matter, and that no longer holds. Cloudflare, Akamai, Fastly, Imperva or any protection placed in front of the server can alter the real access of crawlers, and with agents, search bots, training bots and mixed-use bots the rules will get more complicated. A technical audit should include an explicit review of bot policies at the CDN and WAF. Validating robots.txt is not enough.
Agents, moreover, are a category of their own; what happens when an assistant browses for you we saw in your customer searches once, Google keeps searching for them.
Frequently asked questions
Does Cloudflare block Googlebot by default?
No: Search stays allowed by default. The risk arises when blocking Training, because Cloudflare applies the most restrictive rule to multipurpose bots like Googlebot, unless you configure the exception.
Can I limit AI training without leaving Google?
Yes, with Google-Extended: it limits certain uses in Gemini and Vertex AI without affecting inclusion in Google Search or working as a ranking signal. Cutting Googlebot at the edge to achieve it is too aggressive.
How do I know whether Cloudflare is slowing down Googlebot?
Review Crawl Stats, URL Inspection, Cloudflare Security Events and the 403 or 429 codes on verified crawlers. If the block happens before the origin, server logs may show nothing.
To finish your coffee
The real headline is not that Cloudflare is "going to block Google". It is that a policy designed to regulate how AI systems use your content can close the door on the search engine if it is applied at a layer unable to distinguish the functions of the same crawler. Google tries to separate them with Google-Extended and Cloudflare with a classification by behavior; if both taxonomies don't match, the site owner bears the cost. From now on, auditing crawler access includes robots.txt, CDN, WAF, bot management and edge logs.
If you want me to review how Googlebot reaches your infrastructure, tell me your case.
Sources and further reading
Cloudflare Blog. "Your site, your rules: new AI traffic options for all customers" - https://blog.cloudflare.com/content-independence-day-ai-options/
Cloudflare Developers. "Block AI Bots" - https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/
Cloudflare Press Release. "Cloudflare Allows the Agentic Internet to Flourish with a Simple Philosophy: Your Content, Your Rules" - https://www.cloudflare.com/press/press-releases/2026/cloudflare-allows-the-agentic-internet-to-flourish-with-a-simple-philosophy-your-content-your-rules/
Google Search Central. "AI features and your website" - https://developers.google.com/search/docs/appearance/ai-features
Google Crawling Infrastructure. "Google's common crawlers" - https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
Google Crawling Infrastructure. "Things to Know about Google's Web Crawling" - https://developers.google.com/crawling/docs/about-crawling
Secondary reading: Lalit Patel. "Cloudflare's 15 September AI crawler change could block Googlebot. Here's what to check" - https://lalitondigital.com/blog/cloudflare-ai-crawler-change-googlebot/
