I found out my own site was blocking AI crawlers, and I never chose it
By Allan Leone on
We spent a day making this site legible to answer engines. Then I read the robots.txt actually being served and found half the work was already cancelled by a setting nobody on our side had touched.
We had just finished a pass on structured data and a plain-text summary for language models. Everything checked out locally. Then I fetched the robots.txt that production actually serves, rather than the one in our repository, and found a managed block sitting above ours that disallowed ClaudeBot, GPTBot, CCBot, Google-Extended, Applebot-Extended and several others outright.
Nobody on our side wrote that. It was injected at the edge by our CDN, from a default. The file in version control and the file on the wire were two different documents, and only one of them was being obeyed.
Check the served file, not the source file
This is the whole lesson and it generalises past robots.txt. Anything a CDN can rewrite, a CDN can rewrite without telling you: security headers, cache directives, redirects, bot policy. If your verification reads the repository, you are testing your intentions rather than your deployment.
- Fetch robots.txt from the public URL and read all of it, including anything above your own rules.
- Do the same for your security headers and your canonical tags.
- Do it after every platform change, not only after your own deploys.
Then make it a decision
Discovering the block is not the same as knowing what to do about it. There is a real choice here and it is not obvious in either direction. Cloudflare now splits AI traffic into search, agent and training so you can answer them separately, and new defaults that block training and agent traffic on ad-bearing pages arrive in September.
For a publisher whose revenue is impressions, blocking the crawlers that take the content and return no visitors is defensible. For a studio whose writing is how people find it, being absent from the systems people now ask for recommendations is a strange thing to opt into by accident.
What we settled on
We allow the crawlers that ground answers and cite sources. We are less relaxed about bulk training collection. That is a position, and positions should be arrived at deliberately, not inherited from whatever your infrastructure vendor shipped as a default this quarter.
The uncomfortable part is that we only found it because we were looking. There is no error, no warning and no dashboard entry when your content quietly stops being readable by the systems that increasingly answer questions on your behalf. Go and read your own robots.txt over HTTP. It takes ten seconds and it is not always the file you think it is.
Tags: ai, engineering, business