Last weekend, shared Claude conversations started appearing in Google search results. These weren't private chats accessed without permission. They were voluntarily created share links. But the people who created them didn't expect Google to index them.
The immediate reaction was "bug." The accurate reaction is different: a technical misconfiguration that we encounter regularly in SEO audits at difrnt. The difference between robots.txt Disallow and noindex looks like a minor config detail. In practice, it determines whether a page is visible or invisible in Google.
We've seen the same misconfiguration on sites with significant organic traffic, where marketing teams had no idea about the implications. If you rely on robots.txt to hide pages from search results, there's a good chance you have the same problem without knowing it. Here's what went wrong and what to check on your own site.
What happened with Claude's shared chats
A report published by Search Engine Journal on July 28 documented the technical details. Claude's /share/ pages had two simultaneous and contradictory directives.
First: a robots.txt file blocking all crawlers from the /share/* path. Second: an X-Robots-Tag: none HTTP header, the server-level equivalent of noindex.
The conflict is straightforward: Google cannot read the noindex header if robots.txt prevents it from accessing the page. Think of it as posting a note on a door saying "don't include this room in the building directory," but locking the door before anyone walks by. The note exists. Nobody reads it.
IT consultant Daniel J. Glover independently confirmed the same conflict on July 26. Google spokesperson Ned Adriance clarified that search engines follow site owner controls but don't decide which pages become public. The responsibility for correct configuration stays with the site owner.
What makes the situation even more instructive: the /public/artifacts/ path on claude.ai was not listed in robots.txt, creating a completely different indexing scenario for similar content. Two paths on the same domain, two different configurations, two opposite outcomes in Google.
Disallow is not noindex: two tools, two effects
Robots.txt Disallow tells a crawler: "Don't access this path." The crawler complies and skips the page entirely. It doesn't read its content, doesn't process its headers, doesn't see any directives inside. It's an access directive, not an indexing directive.
Noindex (either as an HTML meta tag or an X-Robots-Tag HTTP header) tells a crawler: "You can come in and read the page, but don't display it in search results." For this instruction to work, the crawler must actually reach the page.
When you combine both, Disallow wins every time. The crawler never reaches the page and never sees the noindex. The page can still appear in the index if Google discovers the URL from other sources: an old sitemap, an external link, a social media share, or even a mention in a public document. Google will display the URL with a message like "No information is available for this page," but the URL remains visible in results.
This isn't the first time it's happened. OpenAI faced a similar situation with ChatGPT transcripts, and Google with Bard conversations. The problem isn't platform-specific. It's a pattern that repeats whenever someone confuses blocking access with blocking visibility.
The situation is especially relevant now that preparing sites for AI crawlers is becoming a priority. If your llms.txt file or site structure is designed for AI accessibility but robots.txt blocks those exact paths, the entire optimization effort is wasted.
Three checks your team can run today
At difrnt. we've previously documented how AI crawlers have already surpassed Googlebot in request volume. The context here is different, but the principle remains: if you're not actively managing how crawlers interact with your site, surprises will follow.
1. Check which pages your robots.txt is blocking. Open Google Search Console and use URL Inspection on pages you want excluded from the index. If you see "Blocked by robots.txt" but the page still appears in results, you have exactly the problem described here. It's a clear sign you need to change your indexing strategy for those URLs.
2. Make sure noindex is accessible to crawlers. Pages that shouldn't be indexed (staging environments, public drafts, internal pages) need noindex without a robots.txt block. If you Disallow the path and add noindex, the latter becomes invisible to Google. You can verify with a simple curl -I on the URL: if you see X-Robots-Tag: noindex in the response, the directive exists. But if robots.txt blocks the path, Google will never make that request.
3. Don't rely on robots.txt for confidentiality. The robots.txt file is public. Anyone can read which paths you've blocked, and ironically, that can draw attention to them. For genuinely sensitive content, use authentication or server-level restrictions, not crawl instructions. Relying on robots.txt for privacy is like hiding a key under the doormat and assuming nobody will look there.
The rule that prevents surprises
If you want a page to stay out of Google, use noindex and don't block it with robots.txt. Let the crawler reach the page, read the instruction, and respect it. It's counterintuitive, but it works precisely because you're granting access to the information you want to communicate: "don't index me."
If you want to save crawl budget for irrelevant URLs (product filters, pagination, sorting parameters), Disallow is the right tool. But don't expect those URLs to vanish from the index if Google already knows about them from other sources.
The combination of robots.txt Disallow + noindex isn't a double layer of protection. It's an instruction that cancels itself out. We've encountered this in audits at companies trying to control AI bot access to their content that found pages they thought were hidden appearing in ChatGPT or Perplexity responses.
The lesson from the Claude incident is clear: technical tools work exactly as documented. Problems arise when we assume they do more than they promise. Robots.txt doesn't promise invisibility. It promises that the crawler won't visit. What the rest of the internet does with your URL is a separate matter entirely. And if that URL circulates on social media, appears in a backlink, or sits in a forgotten sitemap, Google will find it regardless.
