Writing in Search Engine Journal, Helen Pollitt, Head of SEO at Getty Images, breaks down the two ways to block AI crawlers — robots.txt and the server stack — and why they are not equivalent. One is a request; the other is enforcement.
Key facts
- Timing: Available now, configurable at any time.
- Who it hits: Stores whose content or product photography is a core asset and who have decided to keep training crawlers out.
- What to do: Decide first whether to block at all — blocking also costs you AI search citations. If you do block, robots.txt alone barely counts as blocking.
What happened
Blocking via robots.txt works the same way as blocking any bot: name the user-agent and add a disallow rule, e.g. User-agent: GPTBot with Disallow: /, or scope it to a folder such as /products/. It relies entirely on the crawler choosing to obey. Server-level blocking can happen at the origin, the CDN or the WAF. The origin reads the request's IP and headers and applies deny, allow or redirect. A CDN intercepts before the request reaches your server, saving bandwidth, and providers like Cloudflare ship presets that separate search crawlers, agents and training bots, with per-bot tuning. A WAF inspects request behavior rather than headers alone, which makes it the most capable layer for catching crawlers that spoof another user-agent.
What it means for independent store owners
The part people get wrong first: blocking AI crawlers is a trade, not a config task. Some of those bots collect training data, others fetch pages so an AI search product can cite them live — GPTBot and OAI-SearchBot are separate user-agents for exactly that reason. Blanket-blocking usually costs you visibility in AI answers well before it meaningfully reduces training exposure. For most content-led stores, that is a bad trade.
The cases that do justify enforcement are narrower: product photography being scraped at scale, original reviews lifted wholesale, or bot traffic driving up hosting cost. In those situations, remember that robots.txt is a notice, and the crawlers worth worrying about are the ones ignoring it. If you already run a CDN like Cloudflare, its bot category toggles are the highest-value layer for the effort involved; behavioral analysis at the WAF is generally worth paying for only after you've been dealing with persistent spoofed crawling. Before changing anything, read a week of server logs and confirm who is actually crawling you and what they take. Blocking from a generic list is an easy way to cut off the bots that were sending you traffic.