The short answer
Block the crawlers that take your content to train models and give nothing back. Allow the ones that fetch pages to answer a customer's question right now, because those are the ones that can cite you and send a visitor. The decision is per bot, not a single on or off switch, and for most businesses the answer is not "block everything".
That distinction has become urgent because the default changed underneath everyone. Cloudflare, which sits in front of roughly a fifth of the web, began blocking AI crawlers by default for new sites in July 2025, and from 15 September 2026 its defaults also block mixed-use crawlers on any page that carries ads, for new customers, new sites added by existing customers, and all free accounts. Many site owners are now blocking bots they never consciously decided to block.
The three kinds of AI crawler, and why the difference matters
AI companies run several bots that behave very differently. Treating them as one category is what leads to either giving everything away or cutting off the traffic you wanted. There are three jobs worth separating.
- Training crawlers collect pages to improve a future model. Nothing about your site's inclusion is visible, attributable, or immediate. OpenAI's GPTBot and Anthropic's ClaudeBot are the common examples, and Google-Extended is the switch that governs Gemini training rather than Google Search.
- Retrieval or search crawlers build the index an assistant searches when it needs current information. OAI-SearchBot and PerplexityBot sit here. If you are blocked, you cannot be the source of the answer, and the link that would have come with it goes to a competitor.
- User-triggered fetches happen when a person pastes your URL into an assistant and asks it to summarise or compare. ChatGPT-User is the usual agent. Blocking this one means a prospective customer who is actively trying to read about you gets an error.
Most businesses, reasoning it through, land in the same place: training is optional, retrieval and user-triggered fetching are not. That is close to the opposite of the blanket blocking now applied by default in many places.
What blocking actually costs you
The cost is not abstract. Assistants now sit between a customer's question and your website, and the pages they can read are the pages they can recommend. We covered the mechanics of how one question becomes several searches in query fan-out: if a retrieval crawler cannot reach your pricing page, your pricing does not exist in that answer.
The awkward part is that blocking rarely removes you from the conversation. It removes your own words from it. An assistant asked about your company can still assemble a description from directory listings, review sites, marketplace pages, old press coverage, and your competitors' comparison pages. You have not achieved silence, you have achieved a version of your business written by everyone except you.
For businesses whose customers research before buying, that is a bad trade. For a publisher whose archive is the product and whose revenue comes from visits and licensing, the maths is genuinely different, which is why the two groups should stop copying each other's robots.txt.
What Cloudflare changed, and what it means for you
Cloudflare's position has hardened in stages. In July 2025 it flipped the default for new sites to block AI crawlers unless the operator pays or is explicitly allowed. It also launched Pay Per Crawl, a marketplace where a site can charge a bot for access, which it is now extending into a Pay Per Use model that charges when content creates value rather than only when a page is fetched.
The September 2026 change targets what Cloudflare calls mixed-use crawlers: a single bot that does search indexing, AI training and agent fetching at once, so a site cannot allow one purpose without allowing the others. From 15 September 2026, those bots are blocked by default on ad-bearing pages for new customers, new sites, and free accounts. Cloudflare has also said that more than half of AI crawler traffic re-fetches pages that have not changed, which is the bandwidth argument behind the policy.
Two practical consequences. First, if your site sits behind Cloudflare on a free plan, check the bot settings rather than assuming your robots.txt is in charge. Second, if your traffic has been sliding and you have been reading it as an algorithm problem, confirm that the assistants can still fetch you at all before diagnosing anything else — the same advice applies as in why a site stops showing up on Google.
A decision table you can actually apply
Decide by what the bot gives back, not by who operates it. Most businesses can work through this in an afternoon, and the answer stays stable for a year.
| Bot type | Example agents | Typical decision |
|---|---|---|
| Search and retrieval | OAI-SearchBot, PerplexityBot, Google-Extended for grounding | Allow. This is where citations and clicks come from. |
| User-triggered fetch | ChatGPT-User, Claude-User | Allow. A real person is asking about you right now. |
| Training | GPTBot, ClaudeBot, CCBot, Bytespider | Judgement call. Allow if your content is marketing; block if your archive is the product. |
| Unidentified or disguised | Agents that ignore robots.txt | Block at the CDN, since a file cannot stop them. |
If you sell a service, the honest reading is that your website is marketing. Its purpose is to be read, quoted and acted on. Charging a fraction of a cent per fetch for a page whose job is to win a project is optimising the wrong number.
How to set this up without breaking search
The mechanics are less involved than the decision. Three layers, in order of how much they actually enforce.
- robots.txt is where you state the policy. Name each agent explicitly with its own Allow or Disallow block rather than using a blanket wildcard, which is what accidentally blocks retrieval along with training.
- Your CDN or WAF is where the policy is enforced, including against bots that ignore the file. This is also where Cloudflare's defaults live, so it is the first place to look when something unexpected is blocked.
- Page-level signals matter for what is read rather than whether it is read: clear headings, self-contained answers, and the structured data that tells a machine what a page is about.
One thing to avoid: blocking Googlebot by accident while aiming at AI. Google-Extended governs Gemini training and grounding, not classic Search ranking, and blocking Googlebot itself removes you from the index entirely. The two are separate agents for a reason.
The llms.txt question
You will be offered llms.txt as the tidy answer: a file that tells assistants which pages matter. It is cheap to publish and does no harm, but no major assistant has committed to honouring it, and it is not a substitute for a crawler policy or for pages that answer questions properly. Treat it as a hint you leave for the future, not a control you own today.
What we tell clients
Allow retrieval and user-triggered bots, make a considered decision on training bots, enforce it at the CDN, and then spend the remaining effort on being worth citing. Visibility inside AI answers is earned by pages that answer a question completely, in public, with evidence — the same work described in tracking brand visibility in AI search and generative engine optimisation.
And keep a copy of the audience you do not have to negotiate for. Every crawler policy is somebody else's default setting away from changing again, which is the argument for an owned audience that no platform decides for you. If you want a second pair of eyes on your bot rules and what they are costing you, tell us what your logs are showing.