Quick answer: If robots.txt blocks a page, its noindex tag is invisible: crawlers can't read a directive they're never allowed to fetch, so the page can still get indexed through other links. This is why shared AI chatbot conversations and internal search pages keep appearing in Google despite a noindex tag. Fix it with a crawlable page plus an HTML noindex or x-robots-tag header, not a robots.txt block.
Key takeaways
- Robots.txt controls crawling, not indexing. Block a page there and any noindex tag on it is never read.
- A blocked-plus-noindexed page can still get indexed (usually with no description) if any other page links to it.
- Shared Claude and ChatGPT conversation URLs got indexed in Google and Bing this year through exactly this pattern.
- One audited site had roughly 2.5 million indexed URLs against about 2.5 thousand real pages after hijacked internal search pages got indexed.
- The fix: let the page be crawled, and add an HTML noindex meta tag or an HTTP x-robots-tag header instead.
- GPTBot, ClaudeBot, OAI-SearchBot, and PerplexityBot follow the same rules, so legacy robots.txt files can break AI crawl access too.
You added a noindex tag, checked it twice, and the page is still showing up in Google. Or worse, in an AI answer. This is one of the most common technical SEO mistakes, and it's rarely obvious why it happens, because the fix that seems most thorough (block it in robots.txt too) is actually the thing breaking it. This article walks through the mechanism, a live 2026 case where it leaked shared AI chatbot conversations into search results, and the correct fix. It also covers why this now matters for AI crawlers specifically, not just Googlebot, and gives you a 15-minute self-check to run on your own site.
Why a noindex tag stops working when robots.txt blocks the page
A noindex tag stops working the moment robots.txt blocks the crawler from fetching the page it's sitting on, because the crawler never reads a directive it isn't allowed to see. Google's own documentation confirms this directly: robots.txt controls whether a page gets crawled, and noindex controls whether a crawled page gets included in results. Kinsta's technical SEO guide reaches the same conclusion independently. Block the crawl, and the noindex tag on that page might as well not exist.
Here's where it gets counterintuitive. If the page is still linked from anywhere else on the web, a category page, a sitemap, a third-party site, Google can still index it using just the link and any surrounding context, without ever fetching the page itself. You end up with the exact opposite of what you intended: a page you tried to hide twice over, showing up in search with a thin, contextless listing.
The 2026 case: shared AI chatbot conversations got indexed the same way
This exact misconfiguration made mainstream tech news this year. CNET reported, with commentary from technical SEO consultant Glenn Gabe, that shared Claude and ChatGPT conversation URLs were turning up indexed in Google and Bing. The sites hosting those share links had blocked the URL pattern in robots.txt and separately tagged it noindex, the exact broken-protection pattern described above. The noindex was never read, and the share links, discoverable through other pages, got indexed anyway.
The result: private-feeling conversation shares ended up publicly searchable, which is a real trust and privacy problem, not just a crawl-budget inefficiency. It's also a useful wake-up call. Any dynamically generated, linkable URL pattern on your own site (chat transcripts, gated resources, personalized results pages) is exposed to the same failure mode if it relies on robots.txt as its only protection.
When it goes wrong at scale: 2.5 million indexed URLs vs. 2.5 thousand real pages
The downstream risk isn't just a handful of stray URLs; it can escalate into a site-wide quality problem. Glenn Gabe audited a site with roughly 2.5 million indexed URLs against about 2.5 thousand real pages after third parties hijacked the site's crawlable internal search result pages to publish spam content at scale. Because those search-result pages were crawlable (not blocked) but also not properly noindexed, they got indexed in bulk, and the resulting index bloat triggered a site-wide quality downgrade. Search Engine Roundtable's coverage of Google's internal search guidelines confirms this is a known, named failure pattern, not an edge case.
That thousand-to-one ratio is the kind of number that gets a stakeholder's attention fast. It's also a reminder that "blocked" and "protected" aren't the same word. A crawlable page with no noindex is just as exposed as a blocked page with an unreadable one; the two failure modes just get there from opposite directions.
The correct fix: noindex meta tag or x-robots-tag, not a robots.txt block
The reliable fix is to let the page be crawled and put the noindex directive somewhere the crawler can actually read it. Kinsta's remediation guide and Google's own documentation both point to the same two options: an HTML <meta name="robots" content="noindex"> tag for standard web pages, or an HTTP X-Robots-Tag: noindex header for non-HTML files like PDFs or for cases where you can't edit the page's HTML directly. Either way, remove the robots.txt disallow for that path so the crawler can reach the page and read the directive.
The only case where you genuinely want a robots.txt block instead is when you don't want the page crawled at all, and you're also willing to accept that it can still get indexed (usually without a description) if it's linked from elsewhere. If your actual goal is "don't show this in search results," noindex on a crawlable page is the tool. If your goal is "don't let crawlers spend resources on this," robots.txt is the tool. Using both at once on the same path is almost never what you want.
Why this matters more now: AI crawlers inherit the same leak
This mechanism doesn't stop at Googlebot. GPTBot (OpenAI's general crawler), ClaudeBot (Anthropic's crawler), OAI-SearchBot (OpenAI's search-specific crawler), and PerplexityBot (Perplexity's crawler) all respect robots.txt and noindex the same way traditional search crawlers do, which means the exact same misconfiguration can leak a page into AI-crawled indexes or accidentally block a page you wanted AI systems to see. The complication is timing: most robots.txt files predate these crawlers entirely, so nobody has gone back to check whether legacy rules interact correctly with them. AI crawlers also read your site differently than a human visitor does, which is one more reason a robots.txt audit written for Googlebot alone is no longer sufficient.
A robots.txt written five years ago to manage Googlebot's crawl budget has no opinion about GPTBot or PerplexityBot, and a broken noindex-plus-block pattern written for Google will break identically for every AI crawler that respects the same standard. Auditing this is now part of the access layer of AI visibility, not a separate technical SEO task.
How to audit your own site for this in 15 minutes
You can find out if you have this problem faster than it takes to read this article. Run a site: search for your own domain combined with your chat, internal-search, or gated-resource URL patterns (for example site:yourdomain.com "search?q=" or your chatbot's share-link path) and see what turns up that shouldn't be publicly indexed. Then open Google Search Console's Page Indexing report and cross-reference any URL flagged "Indexed, though blocked by robots.txt," a warning documented across multiple SEO troubleshooting guides, against your actual robots.txt disallow list. This is one piece of a broader AI visibility check worth running on the same schedule.
Any URL that shows up in both places, blocked in robots.txt and still indexed, is a confirmed broken-protection case. Fix each one with the correct method from the previous section: unblock the path and add a proper noindex tag or header if the goal was to keep it out of results, or remove the inbound links if the goal was to stop discovery entirely.
A noindex tag only works if the crawler is allowed to read it, and blocking the same page in robots.txt guarantees it never will be. This isn't a rare misconfiguration: it's the exact pattern that leaked shared AI chatbot conversations into Google and Bing this year, and the same pattern that turned one site's internal search pages into 2.5 million spam-indexed URLs. The fix takes minutes once you know what to look for: let the page be crawled, put the noindex where the crawler can see it, and run the site: and Search Console cross-check on your own domain today.
Next step: If you want this checked alongside the rest of your AI-crawler access, not just Googlebot's, Uygen's AI Visibility Audit includes a robots.txt and indexation check as part of the Access pillar.
FAQ
Why is my noindex page still showing up in Google?
The most common cause is that the page is also blocked in robots.txt. When that happens, Google's crawler is never allowed to fetch the page, so it never reads the noindex tag sitting on it. If the page is linked from anywhere else, Google can still index it using the link and surrounding context alone. The fix is to remove the robots.txt block and let the crawler see the noindex directive.
Can Google index a page that is blocked by robots.txt?
Yes. Robots.txt only controls whether Google is allowed to crawl (fetch) a page; it does not remove that page from eligibility for indexing. If another page links to the blocked URL, Google can index it based on that link and any anchor text or surrounding context, typically showing it in results with no description, since Google never actually read the page content.
What does 'Indexed, though blocked by robots.txt' mean in Search Console?
It means Google found and indexed a URL that your robots.txt file disallows crawling. This is usually a sign that a noindex tag on that page (if one exists) was never read, because the crawler wasn't allowed to fetch it. Cross-check the flagged URL against your robots.txt rules and either remove the block or clarify your intent with a properly crawlable noindex.
How do I stop internal search or chatbot conversation pages from being indexed?
Let those pages be crawled, and add an HTML noindex meta tag or an HTTP x-robots-tag header directly on them, rather than blocking them in robots.txt. If you also want to stop them from being discovered at all, remove any internal links pointing to them. Blocking in robots.txt while also relying on noindex is the specific combination that fails.
Does this misconfiguration also affect AI crawlers like GPTBot or ClaudeBot?
Yes. GPTBot, ClaudeBot, OAI-SearchBot, and PerplexityBot all follow robots.txt and noindex directives the same way Googlebot does, so the identical broken-protection pattern can leak pages into AI-crawled indexes or unintentionally block content you wanted AI systems to see. Most existing robots.txt files were written before these crawlers existed, so this is worth auditing specifically, not assuming your Google-focused setup already covers it.
Not sure what else robots.txt might be leaking or blocking?
The AI Visibility Audit checks robots.txt, noindex, and AI crawler access as part of the Access pillar, so you find the misconfigurations before a prospect or a stranger does.