AI crawlers can read your website only if three things are true: your robots.txt and your CDN or firewall let them in, the content is in the HTML your server sends, and the pages load quickly and reliably. Most AI crawlers do not run JavaScript, so content that only appears after scripts run in the browser is invisible to them. You can test all three in about ten minutes.
The crawlers that matter
Each AI company runs several crawlers with different jobs. The important distinction is between crawlers that power search answers, crawlers that collect training data, and fetchers that visit a page when a user asks. They are controlled separately.
| Company | Crawler | What it does |
|---|---|---|
| OpenAI | OAI-SearchBot | Finds pages for ChatGPT search answers |
| OpenAI | GPTBot | Collects content that may be used to train models |
| OpenAI | ChatGPT-User | Visits a page when a ChatGPT user's request needs it |
| Perplexity | PerplexityBot | Builds the index behind Perplexity answers (not used for training) |
| Perplexity | Perplexity-User | Fetches a page when a user asks a question |
| Anthropic | Claude-SearchBot | Indexes content for search results in Claude |
| Anthropic | ClaudeBot | Collects training data |
| Anthropic | Claude-User | Fetches pages for user requests |
Googlebot | Crawls for Google Search, including AI Overviews and AI Mode | |
Google-Extended | A robots.txt token for Gemini training and grounding outside Search | |
| Microsoft | Bingbot | Crawls for Bing, which grounds Copilot's web answers |
Two details cause most mistakes.
First, Google-Extended is not how you control AI Overviews. Google says it does not affect inclusion in Google Search. AI Overviews and AI Mode are part of Search and use the regular Googlebot index. Blocking Google-Extended keeps you out of Gemini training, not out of AI Overviews. Blocking Googlebot removes you from Search entirely.
Second, user-initiated fetchers may ignore robots.txt. OpenAI and Perplexity both say that because a person requested the page, these fetches do not follow the same rules as automatic crawling.
Step 1: Read your robots.txt
Open yourdomain.com/robots.txt. A common setup for a business that wants to appear in AI answers but keep its content out of model training looks like this:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xmlSearch crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot) fall under User-agent: * and are allowed. Whether to block training crawlers is a business decision, not a technical requirement. Our own site allows all of them.
Watch for leftovers: a Disallow: / from a staging site that went live, or a plugin that added blanket blocks. OpenAI and Perplexity say robots.txt changes can take about 24 hours to take effect.
Step 2: Check your CDN or firewall
robots.txt is a request. A CDN or firewall is a gate. Many websites now sit behind services that can block bots before they ever reach your robots.txt.
Cloudflare is the best-known example. In July 2025 it began blocking AI crawlers by default for new domains, and it now manages AI crawler access in its AI Crawl Control settings, where each crawler can be allowed or blocked. If your site uses Cloudflare, open those settings and confirm the search crawlers you want are allowed. Other CDNs and security plugins have similar controls under names like "bot protection" or "bot fight mode".
This is the step most businesses miss, because nothing looks wrong. The site loads normally for people, and crawlers are refused out of sight.
Step 3: Make sure the content is in the HTML
When a crawler requests a page, it receives the HTML your server sends. Browsers then run JavaScript, which on many sites fetches and draws the actual content. A large analysis of crawler traffic by Vercel found that the major AI crawlers download JavaScript files but do not execute them. Google's crawler renders JavaScript for Search, but AI crawlers from OpenAI, Anthropic and others read only the initial HTML.
How to test:
- Open an important page, such as a service page or case study.
- Right-click and choose View Page Source. This shows what the server sent, before scripts ran.
- Press Ctrl+F (Cmd+F on Mac) and search for a sentence from the middle of the page, a price, or a service name.
If you find it, crawlers can read it. If the source is mostly <script> tags and little text, the content is being assembled in the browser.
From a terminal, the same check is:
curl -s https://yourdomain.com/services/ | grep -i "a sentence from the page"Where content goes missing:
- sites built as single-page apps without server rendering
- sliders, tabs and accordions that load their content on click
- "load more" buttons for projects or articles
- reviews and testimonials embedded from a third-party widget
- prices or details loaded from an external system
Standard WordPress pages send finished HTML, which is one reason WordPress sites tend to do well here. On any platform, the fix for missing content is to render it on the server or include it in the page as it loads.
Step 4: Check speed and errors
Crawlers give up on slow or failing pages, and they do not retry forever. Check that:
- key pages load in a few seconds and return a normal status (200), not an error
- old URLs redirect to their new versions instead of breaking
- your XML sitemap lists your current pages and is referenced in robots.txt
Search Console and Bing Webmaster Tools show crawl errors for Google and Bing. Your server or CDN logs show visits from AI crawlers by user agent.
What about llms.txt?
You may have seen advice to add an llms.txt file: a plain-text summary of your site, placed at the root like robots.txt. It was proposed in 2024 as a way to point language models at a site's most useful pages.
It is harmless to publish one, and it can be a useful discipline: writing a one-paragraph summary of what your business does is good practice anyway. But treat it as optional. At the time of writing, the major AI search providers document robots.txt and their crawler user agents as the controls that matter, and none of them has said it uses llms.txt to decide what appears in answers. Getting the four checks above right will do far more.
A note on structured data
Crawlers that can read your HTML can also read the structured data in it. Schema.org markup (usually a JSON-LD block in the page) restates your company name, services, prices and addresses in a format no one has to interpret. It does not unlock access on its own, but once a crawler is in, it removes guesswork. We cover which types matter in the next article in this series.
A ten-minute audit
- Read robots.txt. Are search crawlers allowed?
- Open your CDN or firewall bot settings. Are the same crawlers allowed there?
- View Page Source on three important pages. Is the main text there?
- Check that the sitemap exists and is listed in robots.txt.
- Look at Search Console and Bing Webmaster Tools for crawl errors.
If all five pass, AI crawlers can read your website. What they make of it depends on the content itself: clear, specific pages with consistent facts. That is covered in What Makes a Website Visible to AI Search?.
Frequently asked questions
Will blocking GPTBot remove my site from ChatGPT?
Not from ChatGPT search. GPTBot collects training data; ChatGPT search uses OAI-SearchBot. OpenAI treats them as separate settings, so you can block one and allow the other.
Does blocking Google-Extended remove me from AI Overviews?
No. Google says Google-Extended does not affect Google Search. AI Overviews and AI Mode use the regular Search index built by Googlebot.
Does Googlebot render JavaScript?
Yes, for Google Search. Most AI crawlers do not, so content that depends on JavaScript can be visible in Google but invisible to ChatGPT, Claude or Perplexity.
How do I know if AI crawlers are visiting my site?
Check your server or CDN logs for their user agents, such as OAI-SearchBot, PerplexityBot or ClaudeBot. Cloudflare's AI Crawl Control dashboard shows this for sites on Cloudflare.