You have a list of AI bots you block via robots.txt. You update it every few weeks. Maybe you added GPTBot last month, ClaudeBot the month before. You feel like you're in control.
You're not. That list isn't a strategy. It's a defensive reflex, written in a hurry, without thinking through consequences. And those consequences vary enormously depending on which bot you block and what that bot actually does with your content.
At difrnt., we manage SEO infrastructure for clients with serious traffic volumes. We moved past the "block everything that sounds like AI" phase a year ago. Here's what we've learned since.
Not all AI bots want the same thing
We treat all AI bots as a single threat. But the difference between a training crawler and an autonomous agent is the same difference between a street photographer and a customer walking into your store.
A training crawler (GPTBot, CCBot, Bytespider) scrapes content at scale to train language models. Once your data enters a training set, you lose control. You get nothing back. The decision to block it is relatively straightforward.
A retrieval bot (ChatGPT-User, PerplexityBot, Google-Extended for AI Overviews) does something entirely different: it fetches content in real time to build answers for users. Block it, and you vanish from those answers. Not tomorrow. Immediately. Semrush data from 2026 shows that sites blocking retrieval bots lose an average of 12% visibility in AI-augmented search results.
Then there are AI agents. These navigate your site like a human user: comparing products, checking prices, reading specs. Cloudflare reports that AI agent traffic grew by over 300% in the past 12 months. You can't easily block them without also blocking real users.
A recent article on Search Engine Journal frames this problem clearly: SEO teams need bot policies, not lists. And from experience, I can confirm it's a shift many are postponing but nobody can afford to ignore.
Robots.txt doesn't decide anything. Your CDN does.
Robots.txt is a voluntary protocol. Period. A bot that chooses to respect your rules will. One that doesn't, won't. There's no enforcement mechanism at this level.
Real access decisions happen at the CDN, firewall, or bot management layer: Cloudflare, Akamai, Fastly. And here's a problem we see at nearly every enterprise client: the SEO team updates robots.txt, but the development team configures the CDN separately. The two don't talk to each other.
The result is predictable: contradictory rules. You block GPTBot in robots.txt, but Cloudflare lets it through. Or the opposite: you allow a retrieval bot in robots.txt, but the WAF classifies it as a threat and blocks it. We had a client losing AI visibility for three months without knowing. The cause? A firewall rule that blocked PerplexityBot as "suspicious automated traffic."
We wrote previously about the limits of robots.txt in the AI era. The conclusion still holds: robots.txt is a signal of intent, not a control mechanism. And signals without real implementation are just wishes.
AI agents are visitors you can't afford to ignore
AI agents are the most complicated category and the fastest growing one. They behave like human users: they have sessions, navigate between pages, interact with interface elements. Some even fill out forms.
Gartner estimates that by 2028, 25% of web searches will be initiated by AI agents, not humans. If you block the agent doing product research for a user, your product disappears from the final recommendation. This isn't a theoretical risk. It's a customer you'll never know existed.
The additional complication: AI agents show up in analytics as normal traffic. Their user agent resembles a standard browser. They're hard to distinguish without advanced fingerprinting solutions or behavioral pattern analysis. We covered how AI agents are changing the rules of search in an earlier article, and the situation has only grown more complex since then.
Our approach? We don't try to block them. We try to understand what they're doing on the site. An agent performing price comparisons for a user is, in practice, a qualified lead with purchase intent. Block it, and you lose exactly the type of visitor you're looking for.
From lists to policies: how it works in practice
We replaced the block list with a decision framework that we apply to every client. It's not flashy, but it works. Here are the steps.
Classify bots by category. Training crawlers, retrieval bots, AI agents, traditional SEO crawlers. Each category gets a separate policy. Not a single "allow" or "disallow."
Evaluate the impact of blocking. For each category, answer concretely: "If we block this bot, what do we lose?" If the answer includes "visibility in AI Search" or "presence in AI recommendations," that block requires explicit approval, not just a line in robots.txt.
Align your teams. SEO, development, security, and legal need to see the same policy and understand why it exists. We've seen cases where the security team blocked a retrieval bot without knowing the SEO impact. A quarterly alignment meeting prevents these incidents.
Review every quarter. The landscape shifts fast. New bots appear monthly. A policy written six months ago is almost certainly outdated. We previously detailed the real cost of AI bot traffic on servers, and one key takeaway is that these decisions need regular revisiting.
Document everything. Every block or allow decision, with its justification. When a new bot appears, you have a decision framework ready. No more improvising under pressure.
What actually matters
Blocking AI bots isn't inherently good or bad. It's a business decision that should be made with data and context, not out of panic or inertia. The block list you have right now probably contains both sound decisions and costly mistakes. The problem is you don't know which is which.
The question isn't "who do I block?" It's "what do I lose if I block?" And if you can't answer that for every bot on your list, you need a process, not a longer list. Start with classification. Continue with impact assessment. And make sure the teams actually implementing the rules know what they're blocking and why.
