There's a file on your website you've probably never opened. It's between 5 and 20 lines long, it sits in the root of your domain, and until recently, it was entirely the developers' responsibility. It's called robots.txt.
Here's the problem: this file now decides who gets to read and use your content. Not just Googlebot, which indexes pages for search results. I'm talking about the 30+ AI crawlers that access your site daily, each with a different purpose: some index you, some cite you in answers, some take your text to train language models.
If you haven't checked your robots.txt in the past 6 months, you've already made a decision. You just didn't make it yourself.
It's Not Just About Googlebot Anymore
For 20 years, robots.txt was straightforward. You had Googlebot, Bingbot, a handful of niche crawlers. You told them what to index and moved on. A developer would add a few lines at deployment, nobody thought about it again, and everyone was happy. That simplicity is gone.
Now, alongside classic search crawlers, your site receives visits from GPTBot (OpenAI), ClaudeBot (Anthropic), Bytespider (ByteDance), CCBot (Common Crawl), Google-Extended, and at least 20 other automated agents. According to the Imperva Bad Bot Report 2024, 49.6% of all web traffic comes from bots. And the share of AI crawlers grows visibly quarter over quarter.
The critical distinction: a search crawler helps you get found. An AI agent crawler uses your content as a source for real-time answers. An AI training crawler takes your content and turns it into raw material for language models. No attribution, no link-back, no traffic.
All three request access through the same file. And if you don't make the distinction explicitly, you treat them all the same.
Three Crawler Categories, Three Different Decisions
From our experience at difrnt. running technical audits and site configurations, we've arrived at a simple working framework. We split crawlers into three categories, each with its own access logic:
Search (Googlebot, Bingbot, Yandex): full access. You want to be indexed, you want to appear in organic results. You don't restrict anything here, except maybe duplicate content pages or admin zones.
AI Agents (ChatGPT browsing, Perplexity, Google AI Overviews): selective access. These crawlers use your content in real time as a source for answers. Block them and you vanish from AI answers. Allow everything and you risk users getting the full answer without ever visiting your site. It's a trade-off that depends on your business model: if you monetize through traffic, think carefully. If you monetize through authority, open the gates.
Training (GPTBot in training mode, CCBot, Bytespider): access that warrants careful evaluation. These crawlers collect content to train language models. You don't get direct traffic in return. But there's a long-term argument: a model trained on your content might recommend you more frequently in its answers. How much that's worth in practice is hard to quantify today.
The real issue is that robots.txt can't make these distinctions with enough granularity. You can block a user-agent entirely, but you can't say "allow browsing, block training" for the same crawler. Google-Extended, for instance, controls both AI training and AI Overviews. Block one, you block both. And that's a decision with real impact on your visibility.
What Robots.txt Doesn't Do (and What Everyone Assumes It Does)
Robots.txt is a goodwill protocol. It's not a firewall. It's not a legal contract. It's a convention that crawlers voluntarily respect.
According to an Originality.ai study, roughly 35% of AI crawlers don't follow robots.txt directives. That means you can set "Disallow" for every AI bot on the list and still have crawlers regularly accessing your content. The file declares what you want, but it doesn't guarantee it happens.
That's why, for the clients we manage, robots.txt is just the first layer. On top of it, we recommend a few concrete practices:
Active server log monitoring: check exactly which crawlers access your site, how often, and which pages they read. Most site owners have never done this exercise. It's like having an open door without knowing who walks in.
Edge-level rules: Cloudflare, AWS CloudFront, Vercel, all offer traffic filtering options before requests reach your server. That's where you apply real blocks, not just advisory notices that crawlers can ignore.
Quarterly review: new crawlers appear constantly, and existing ones change their user-agent strings. An outdated robots.txt may block access you actually want or allow exactly what you don't.
Decision documentation: like any business decision, every rule needs a "why" behind it. Six months from now, nobody will remember the reasoning behind a line added in a Friday evening commit.
When a Text File Becomes a Strategy Conversation
Last month we had completely opposite conversations with two clients.
The first, an e-commerce site with over 40,000 products, wanted to block all AI training but allow AI Overviews. The logic: "I want to appear in AI answers, but I don't want models copying my product descriptions and serving them to competitors." The problem: blocking Google-Extended eliminates both functions. They had to choose, and they chose visibility. But it was an hour-long discussion with arguments on both sides, not a 5-second commit.
The second, a B2B content publisher, wanted the exact opposite: "We want AI to be trained on our content. We want to become the source it cites." They opened full access and invested in structured data to maximize visibility in AI answers.
Both decisions are valid. Both involved marketing, management, and technical teams. And both started from a question every business should ask: who do we want to have access to our content, and under what conditions?
Robots.txt is no longer a technical artifact. It's a content access policy. And in 2026, that policy matters as much as your SEO strategy or your content marketing budget. If you haven't discussed it as a team, you probably have a default policy you never consciously chose.
Open the file. Read it. And ask yourself one question: is this what I wanted? If the answer is "I don't know," that's the problem.
