Home/News/Robots.txt Only Asks: Blocking AI Crawlers Takes CDN or WAF Rules

SEO & GEO

Robots.txt Only Asks: Blocking AI Crawlers Takes CDN or WAF Rules

2026-09-03 Search Engine Journal SEO & GEO
Robots.txt Only Asks: Blocking AI Crawlers Takes CDN or WAF Rules
图:Search Engine Journal

Writing in Search Engine Journal, Helen Pollitt, Head of SEO at Getty Images, breaks down the two ways to block AI crawlers — robots.txt and the server stack — and why they are not equivalent. One is a request; the other is enforcement.

Key facts

  • Timing: Available now, configurable at any time.
  • Who it hits: Stores whose content or product photography is a core asset and who have decided to keep training crawlers out.
  • What to do: Decide first whether to block at all — blocking also costs you AI search citations. If you do block, robots.txt alone barely counts as blocking.

What happened

Blocking via robots.txt works the same way as blocking any bot: name the user-agent and add a disallow rule, e.g. User-agent: GPTBot with Disallow: /, or scope it to a folder such as /products/. It relies entirely on the crawler choosing to obey. Server-level blocking can happen at the origin, the CDN or the WAF. The origin reads the request's IP and headers and applies deny, allow or redirect. A CDN intercepts before the request reaches your server, saving bandwidth, and providers like Cloudflare ship presets that separate search crawlers, agents and training bots, with per-bot tuning. A WAF inspects request behavior rather than headers alone, which makes it the most capable layer for catching crawlers that spoof another user-agent.

What it means for independent store owners

The part people get wrong first: blocking AI crawlers is a trade, not a config task. Some of those bots collect training data, others fetch pages so an AI search product can cite them live — GPTBot and OAI-SearchBot are separate user-agents for exactly that reason. Blanket-blocking usually costs you visibility in AI answers well before it meaningfully reduces training exposure. For most content-led stores, that is a bad trade.

The cases that do justify enforcement are narrower: product photography being scraped at scale, original reviews lifted wholesale, or bot traffic driving up hosting cost. In those situations, remember that robots.txt is a notice, and the crawlers worth worrying about are the ones ignoring it. If you already run a CDN like Cloudflare, its bot category toggles are the highest-value layer for the effort involved; behavioral analysis at the WAF is generally worth paying for only after you've been dealing with persistent spoofed crawling. Before changing anything, read a week of server logs and confirm who is actually crawling you and what they take. Blocking from a generic list is an easy way to cut off the bots that were sending you traffic.

Someone should be watching these changes for you

I run AI automation for independent stores — content, SEO/GEO, product listing, social, and keeping up with policy shifts like this one.

See how it works