If you work in content marketing, you have probably heard the recommendation: create an llms.txt file to be visible to AI. Add it to your site, optimize it, make sure AI bots understand your content. Sounds logical. In a world where ChatGPT, Gemini, and Perplexity are taking over functions traditionally handled by search engines, preparing for AI Search seems urgent.
Except the data tells a different story. A study recently published by Ahrefs, analyzing 137,000 domains, shows that 97% of llms.txt files received zero requests from any bot. None. And among those that did receive traffic, most came from SEO audit tools, not AI engines.
As Head of Content at difrnt., I spend a lot of time thinking about which content investments actually pay off. And the Ahrefs study published on Search Engine Journal raises a question every content team should be asking: are we optimizing for AI, or optimizing for an idea of AI?
What is llms.txt and where did the hype come from
llms.txt is a text file placed at the root of a website (similar to robots.txt) that provides instructions and context for language models. The idea: if an LLM visits your site, llms.txt tells it what matters, how the content is structured, and where to find key information. Think of it as a site map written specifically for AI.
The proposal gained traction quickly in the SEO and GEO (Generative Engine Optimization) community as a proactive step toward AI visibility. The logic seemed solid: robots.txt controls traditional crawlers, llms.txt does the same for AI bots. Many marketers added it to their checklist without verifying whether it actually works. Consultants included it in standard audits. Agencies added "llms.txt implementation" as a line item in proposals.
The problem? Only 28% of the 137,000 domains analyzed by Ahrefs had published an llms.txt file. And of those roughly 38,000 valid files, only about 1,100 received any traffic at all. Real-world usage is practically nonexistent.
What the Ahrefs data actually shows
The numbers are worth examining in detail because they dismantle several assumptions. Of all requests to llms.txt files:
21% came from SEO audit tools (not from AI). The most active "reader" of llms.txt files turns out to be other marketing tools checking whether you implemented it correctly. An irony worth noting: the file created for AI is read mostly by tools that check if you created it.
13% came from traditional web crawlers, including Googlebot. Google scans the file, but not for AI Overviews or SGE. It is a standard indexing check, the same way it scans any file in a site root directory.
AI bots generated 19% of requests, but the breakdown matters: 10% are coding agents (GitHub Copilot, Cursor, and similar programming tools), 5% are training crawlers (data collection for model training, not for citations), and only 2% are AI assistants (Perplexity, ChatGPT browsing). In other words, the AI assistants that generate citations and recommendations, the exact reason you created the file, barely access it.
One detail that says a lot: Slackbot fetched llms.txt files more frequently than PerplexityBot. When an internal messaging platform reads your "AI-optimized" content more often than the AI engine that is supposed to cite it, something is not working as expected.
Google’s John Mueller was direct: llms.txt "is not done for search" and described it as "a temporary crutch, perhaps to save some tokens" used by coding tools.
The gap between what we prepare and what works
The problem is not that llms.txt is a bad idea. It may become useful in the future as the AI ecosystem matures. The problem is about priorities. Content teams have limited time and resources, and every hour invested in a format nobody reads is an hour not invested in content that generates actual results.
From our experience at difrnt., AI selects citations based on clarity and specificity, not based on a configuration file. An article that directly answers a question with concrete data and specific entities (Google Analytics 4, not "an analytics tool") has a much better chance of being cited by AI than a site with perfect llms.txt but generic content.
We already know what works for visibility in AI Search, and it is no secret:
Content with clear structure. Paragraphs that directly answer specific questions are preferred by AI engines. This is not about keyword stuffing. It is about organizing information into logical blocks that a language model can extract and cite accurately.
FAQs with direct answers. Question-and-answer sections remain the most cited content format in AI Search. AI prefers concise, to-the-point answers that can be included as-is in a generated result.
Specific entities instead of generic terms. Mention Meta Ads Manager, not "advertising platforms." Talk about Semrush or Ahrefs, not "SEO tools." Specificity helps AI connect your content to real user queries.
Where we invest as a content team
If you already have an llms.txt file published, do not delete it. It is low-effort and does no harm. But if you are just now planning your content strategy for AI visibility, here is what we prioritize at difrnt.:
First investment: original content with a unique perspective. AI prefers unique sources with opinions and data not found in 50 other articles. AI bots consume server resources, but citations come when you have something to say that nobody else is saying.
Second: updating existing content. A 2024 article with 2023 data loses relevance not just for Google but also for AI. Language models have a documented preference for recent sources. Regularly updating content that already performs well offers clearer ROI than creating new configuration files.
Third: distribution on channels where AI collects data. Reddit, niche forums, professional communities on LinkedIn. AI does not only read websites. It reads discussions from social and community platforms. Presence in real conversations matters more than any configuration file.
97% of llms.txt files are ignored not because the idea is wrong, but because the ecosystem is not there yet. In the meantime, the best content strategy for AI remains the same: write clearly, be specific, offer a unique perspective. Which, not coincidentally, works for humans too.
