Your robots.txt is Probably Blocking ChatGPT Right Now
Open a new tab and go to yourdomain.com/robots.txt. If you see Disallow: / under User-agent: * and no specific Allow rules for AI crawlers, ChatGPT and Perplexity cannot read your site.
Here is a mistake that costs founders AI visibility every single day: they set up robots.txt years ago - possibly using a WordPress plugin default, a developer template, or a Shopify preset - and never thought about it again. That file is now silently blocking every AI crawler that visits their domain.
In GeoIQ audits, we find robots.txt blocking issues in roughly 30% of the Indian startup sites we check. Some block AI crawlers intentionally (a valid choice). Most block them by accident - a legacy Disallow: / rule under User-agent: * that was meant to keep staging content out of Google, but now catches everything including GPTBot, PerplexityBot, and ClaudeBot.
What AI crawlers are and why they matter
Each major AI system has its own web crawler that reads your site to build knowledge of your brand:
Crawls for ChatGPT training data and knowledge updates. Blocking this means ChatGPT cannot learn about new content on your site.
Used when ChatGPT's browsing mode searches the web in real time. Blocking this means ChatGPT cannot cite your pages in live responses.
Perplexity crawls the live web in real time. This crawler is how Perplexity finds your content and cites your brand in answers.
Claude's web crawler for training data and knowledge. Blocking this makes your site invisible to Claude's training pipeline.
Google's crawler specifically for Gemini training data. Separate from Googlebot - blocking Googlebot does NOT block Google-Extended.
GeoIQ checks your robots.txt, llms.txt, Organization schema, and 6 AI systems. 60 seconds. No signup.
How to check your robots.txt in 30 seconds
Go to yourdomain.com/robots.txt in your browser. You will see the raw text content. Here is what to look for:
User-agent: * Disallow: / # AI crawlers cannot get past this # Even if GPTBot is not mentioned, the Disallow: / above blocks it
User-agent: GPTBot Disallow: / User-agent: PerplexityBot Disallow: /
User-agent: * Disallow: /wp-admin/ Disallow: /staging/ # Looks fine... but if you have a Noindex directive on most pages, # AI crawlers see the pages but cannot cite them
The fix: exact rules to add
Here is the minimal robots.txt that correctly handles all major AI crawlers. Copy this and adjust for your specific needs:
User-agent: * Allow: / Disallow: /admin/ Disallow: /account/ Disallow: /checkout/ # Allow all major AI crawlers explicitly User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Google-Extended Allow: / # Sitemap location (required) Sitemap: https://yourdomain.com/sitemap.xml
Replace /admin/ and /account/ with whatever directories you legitimately want to keep private. Keep the AI crawler sections exactly as shown.
If you have a WordPress site
WordPress sometimes auto-generates robots.txt rules, especially if "Discourage search engines" is checked in Settings under Reading. Check that first. If you use a plugin like Yoast SEO or Rank Math, those tools manage robots.txt - find the robots.txt editor inside the plugin settings and add the AI crawler rules there.
# AI crawler allow rules - add to your existing robots.txt User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Google-Extended Allow: /
Case study: MealCoreAI
MealCoreAI is a health app for Indian users managing diabetes. When we ran a GeoIQ audit in May 2026, the AI audit flagged two issues: no llms.txt and no explicit AI crawler rules in robots.txt. The site was not actively blocking crawlers (no Disallow: /), but it also was not explicitly allowing them.
After adding explicit Allow rules for all five major AI crawlers and creating a basic llms.txt, GeoIQ's Perplexity sub-score improved from 18 to 30 within three weeks. ChatGPT and Gemini scores remained near zero - they rely on training data, not live crawls, and that takes longer to update. But the live-web component improved immediately.
The lesson: robots.txt fixes are the fastest-acting technical GEO change you can make. Perplexity responds within 2-4 weeks of new crawler access. ChatGPT training data cycles more slowly, but you cannot improve that score at all if you are blocking GPTBot.
GeoIQ checks your robots.txt, llms.txt, Organization schema, and 6 AI systems. 60 seconds. No signup.
What to do after fixing robots.txt
Fixing robots.txt removes a blocker. It does not directly add citations or tell AI systems what your brand is. After the fix, do these two things to compound the benefit:
Create llms.txt. Place a plain text file at yourdomain.com/llms.txt that describes your brand to AI crawlers. When GPTBot and PerplexityBot visit your site and find a well-written llms.txt, they get structured brand context without needing to parse dozens of pages. See our llms.txt complete guide for the exact format.
Run a full GeoIQ audit. The audit checks your robots.txt, llms.txt, Organization schema, AI crawler accessibility, and your live scores across 6 AI systems. It tells you exactly what is working and what to fix next, ranked by impact.
Frequently asked questions
No. GPTBot is OpenAI's crawler, separate from Googlebot. Blocking GPTBot has no effect on your Google rankings. It only prevents OpenAI from crawling your site for ChatGPT training data and browsing mode.
The main ones: GPTBot (OpenAI/ChatGPT), PerplexityBot (Perplexity), ClaudeBot (Anthropic/Claude), Google-Extended (Gemini training data), OAI-SearchBot (ChatGPT browsing). Also worth adding CCBot and ChatGPT-User for completeness.
No. If a user-agent is not mentioned in robots.txt, it follows the rules for User-agent: * (the wildcard). If you have Disallow: / under User-agent: *, all crawlers including AI ones are blocked. If you have Allow: / under User-agent: *, all crawlers are allowed unless specifically restricted.
That is a valid choice but comes with a cost: you become invisible in AI recommendations. If your business benefits from being discovered via ChatGPT or Perplexity answers, blocking AI crawlers removes that channel entirely. If content protection is more important than AI discovery, blocking is reasonable.
Visit yourdomain.com/robots.txt in your browser. Look for 'PerplexityBot' - if it is not listed, check the User-agent: * rules. If those have Disallow: /, PerplexityBot is blocked. Add User-agent: PerplexityBot followed by Allow: / to explicitly allow it.
It opens the door but the effect takes time. Once AI crawlers can access your site, they still need to crawl it, index it, and either incorporate it into training data (slow) or surface it in live-search mode (faster, 1-4 weeks for Perplexity). Think of it as removing a blocker rather than an instant score booster.