If you have been in digital marketing long enough, you have probably noticed something new happening. Companies are no longer just optimizing for Google. They are building separate content dedicated entirely to AI models: llms.txt files, markdown pages invisible to regular visitors, endpoints that serve information only machines will ever see.
It sounds smart. AI is becoming a discovery channel, and you need to be present. But for anyone who lived through the SEO cycles of the past 15 years, the scenario feels uncomfortably familiar. We have built a parallel web for machines before. It did not end well.
A recent article on Search Engine Journal names the phenomenon directly: the shadow web. And it raises a question too few people are asking: what happens when the information humans receive is different from what AI models receive?
Why brands are building a second web
The motivation is pragmatic, and that is exactly what makes it tempting. Publishing on the main site is slow. The CMS has technical constraints, legal needs to approve, design has to adapt, and the dev team's backlog is full through next month. Meanwhile, ChatGPT, Gemini, and Perplexity want clear, structured answers right now.
So brands take a shortcut. They publish an llms.txt file that guides AI models toward clean content. They create separate markdown pages, invisible to regular site visitors. Cloudflare launched a dedicated service, Markdown for Agents, that serves alternative versions of existing pages. New APIs and informational feeds are appearing, built exclusively for machine consumption.
Each initiative has a reasonable justification on its own. But taken together, they create a content layer that exists only for digital eyes. A shadow web no human user will ever see.
Same instincts, different packaging
If you have done SEO for over a decade, the term "cloaking" is not new to you. In the early 2000s, the industry built content at scale that users never saw: hidden text, doorway pages, systematically manufactured links. Search engines responded methodically, but it took them years to clean up the mess.
In 2011, Google Panda hit low-quality content created solely for ranking. A year later, Penguin targeted systematic link spam. In 2022, the Helpful Content system formalized a rule everyone knew but kept ignoring: content must be made for people, not algorithms. And in 2024, updated spam policies explicitly target content generated at scale, whether written by humans or AI.
The fundamental rule has not changed in 20 years: what you show the machine must be what you show the user. Google still classifies cloaking as spam, with no technical or semantic exceptions. Yet here we are, building the exact same separation again. We no longer say "content for Google." We say "content for AI." The terminology changed. The logic is identical.
When reformatting becomes manipulation
There is an important distinction worth making explicit. Presenting the same information in a format that is easier for machines to process is not manipulation. Schema markup, structured data, XML sitemaps, FAQ sections with direct answers: these all help AI understand what you have already published for people. It is transparent, verifiable, and useful to both audiences.
The problem starts when you create different information for different audiences. When your page says one thing but your llms.txt file says another. When you publish details in the AI feed that cannot be found anywhere on the site. When the machine version contains promises or specifications that the human version does not support.
The stakes are growing fast. According to data published by Search Engine Journal, roughly 1 in 5 consumer messages on ChatGPT now seek factual information, not text generation. AI has become a discovery channel with a steadily growing audience. If your brand sends different information through the human channel and the AI channel, the discrepancy will become visible sooner than you think.
AI providers themselves have no interest in legitimizing a system they would then have to police. OpenAI, Google, and Anthropic need accurate answers to maintain user trust. If brands begin manipulating information served exclusively to AI, these platforms will respond exactly as search engines did: with filters, penalties, and exclusions.
We wrote about llms.txt and its real-world adoption a few weeks ago. The conclusion from that piece still holds: an extra file does not fix a content problem. But it can make things worse if it introduces a second version of the truth.
One content source, for everyone
The answer is not to give up on AI visibility. It is to stop building a second web to achieve it.
In practice, this means a few concrete things. Structure your existing content with clear semantic markup. Implement structured data on every product, service, and article page. Add FAQ sections that are visible to both people and AI models, with direct answers to the questions a potential customer would ask. Name specific entities: Google Analytics 4, not "an analytics tool." Meta Ads Manager, not "an advertising platform."
Think about structured data, XML sitemaps, and product feeds. None of these technologies have ever been controversial, because they serve the same information in a different format. That is exactly what AI optimization should look like: a technical layer on top of existing content, not a parallel content layer.
If an AI model cannot extract the right information from your site, the problem is not a missing file. The problem is your site's structure and clarity. And that is actually good news, because the fix serves both audiences at the same time.
A shadow web also comes with an operational cost that few organizations calculate upfront. It needs to be maintained, updated, and governed separately. When you change prices on the site, do you also change them in the AI feed? When you update a product specification, do you also update the markdown page? Most companies barely manage to keep one site accurate. Two sources of truth do not mean more visibility. They mean more opportunities for contradiction.
The pressure to be visible in AI search results is legitimate. But the shortcut through invisible content is exactly the kind of optimization that search engines spent 15 years penalizing. Make your site better for everyone. Do not build a second site just for machines.
