Robots.txt Was Never Built for This

Robots.txt was invented in 1994. A plain text file in your site's root directory that told crawlers where not to go. Admin pages. Test files. Internal directories. It worked because there were few crawlers and they all played by the same rules.

Thirty years later, that same file is being asked to do something entirely different: control which AI models can read, train on, and redistribute your content. And no, it was never equipped for that.

At difrnt., we started getting questions about robots.txt and AI around 2024. Initially they were simple: "Can we block ChatGPT?" Now they're far more nuanced. And the answer is almost never a straight yes or no.

When a 1994 File Becomes AI Traffic Police

Robots.txt was designed with a clear premise: crawlers would voluntarily respect the rules. There's no enforcement. It's a gentleman's agreement, not a firewall. Googlebot respected robots.txt for 25 years because it had an interest in being a good citizen of the web. So did Yahoo, Bing, and the rest.

But the crawler universe has expanded dramatically. Industry data shows that AI bot traffic on websites grew by over 300% in the past year. GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended, CCBot, Bytespider, Amazonbot, each arrives with its own user-agent and its own interpretation of the rules.

The problem? Each AI company independently chose how to name its crawler. There's no standard. If you want to block AI training on your content, you need to maintain a list of user-agents that grows monthly. Right now, a comprehensive robots.txt covering the major AI crawlers has over 30 lines dedicated solely to this purpose.

We've seen this firsthand in the technical audits we run. One e-commerce client had 14 distinct AI crawlers in their server logs, of which only 3 were mentioned in their robots.txt. The rest accessed the site completely unimpeded. Not because the owner didn't want to block them, but because they simply didn't know these crawlers existed.

Blocking AI Doesn't Mean What You Think It Means

The most common misconception we encounter with clients: "I blocked GPTBot, so ChatGPT knows nothing about us." Wrong. On multiple levels.

First, robots.txt controls crawling, not training. If an AI model was trained on data from Common Crawl or other public datasets that included your site before you blocked the crawler, that content is already in the model. Blocking is prospective, not retroactive. A 2025 report estimated that over 60% of the public web was included in training sets for major language models. The probability that your site is already in there is high.

Second, there's a critical difference between training and grounding (or retrieval). When you block Google-Extended, you block Gemini model training on your content. But you don't necessarily block Gemini from citing or summarizing your content in AI Overviews, because Google uses its search index for that, not the training crawler. Same content, two completely different uses, one control mechanism.

This creates an interesting paradox. Many site owners want to protect their content from AI "theft," but at the same time want to appear in AI responses. It's like wanting to block a restaurant from copying your recipe while simultaneously asking them to recommend you to their customers. The two goals can be contradictory, and robots.txt doesn't have the granularity to separate them cleanly.

Third, some companies use multiple user-agents for different purposes. Anthropic, for example, has ClaudeBot for training, Claude-User for direct interactions, and Claude-SearchBot for search. OpenAI has GPTBot for training and OAI-SearchBot for search. If you block GPTBot but not OAI-SearchBot, your content can still be accessed and used in certain contexts. The correct configuration depends on exactly what you want to control.

Standardization Is Coming, But Slowly

The good news: the industry is starting to recognize the problem. Google, Microsoft, and OpenAI are working on a shared opt-out mechanism that would allow a single robots.txt line to communicate training preferences to multiple providers simultaneously. Anthropic is also in discussions. The idea is to replace 30 separate rules with one universal one.

The less good news: full implementation is estimated for 2027. Until then, each provider will continue interpreting rules differently. Microsoft plans to replace the NoArchive and NoCache directives with the new system only from March 2027. Google will treat the new option "as if" you had blocked Google-Extended, which creates ambiguities for sites that want to allow grounding but not training.

In practice, this means robots.txt for AI will remain complicated for at least another year. And even after standardization, you'll need to verify that each provider honors the same rules in the same way. Not even a universal standard eliminates the need for monitoring.

What We Recommend at difrnt.

We've developed a checklist that we apply in all technical audits. It's not flashy, but it works. A few essential points:

1. Take inventory before you block. Before adding rules to robots.txt, check your server logs. How many AI bots actually access the site? How often? How much bandwidth do they consume? Many clients want to block crawlers that don't even visit their site. Other times, we discover crawlers generating 40-50% of total traffic, consuming resources with zero benefit.

2. Decide what you want to control. Do you want to block training? Citations in AI responses? Both? The answer determines the configuration. For most businesses, visibility in AI responses is an advantage, not a threat. Total blocking means disappearing from AI Search, and that's a business decision, not a technical one.

3. Monitor after implementation. Robots.txt has no delivery confirmation. You don't know if a crawler respects the rules unless you check the logs. We verify 30 days after any change and have found cases where smaller crawlers completely ignored directives. Without monitoring, robots.txt is just an act of hope.

4. Don't forget sitemap.xml. A restrictive robots.txt combined with a permissive sitemap.xml sends contradictory signals. If you block a crawler in robots.txt but invite it through the sitemap, behavior varies from provider to provider. Some ignore it, others interpret it as an explicit invitation that overrides the restriction.

5. Review quarterly. The list of AI crawlers changes constantly. New user-agents appear, some get renamed, others are replaced. A robots.txt that was complete in January may be outdated by April. We've added robots.txt updates as a fixed item in the quarterly technical reviews we run for clients.

Your 1994 text file isn't going away anytime soon. But its role is fundamentally changing. It's no longer about "don't come here." It's about "who gets to do what with my content." And for that, a static text file is an ill-suited tool used for lack of something better.

When big brands optimize their AI Search presence, smaller ones need to at least understand what controls they have available. Even if those controls are imperfect.