The AI crawler list: GPTBot, ClaudeBot, PerplexityBot and how to allow them

AI crawlers are bots that AI companies use to read web pages, either to answer users' questions with up-to-date sources or to collect training data for future models. They identify themselves with a user-agent name and, at the major vendors, follow the rules in your robots.txt file (RFC 9309). Some vendors note that fetches triggered directly by a user may be handled differently, so read each vendor's page.

The important thing is that one company usually runs several bots with different jobs. Blocking "OpenAI" in one line can remove you from ChatGPT search, even if all you wanted was to opt out of training.

The table lists the user-agent tokens that the major vendors document. Each vendor's own page is linked under Sources.

Allow every search crawler and user agent if you want AI engines to recommend and cite you. These bots are how ChatGPT, Claude and Perplexity read your pages when they answer a question (OpenAI, Perplexity). Blocking them doesn't stop the assistant from talking about you; it only stops it from reading your version.

Training crawlers are a business decision. Allowing them helps future models learn accurate facts about your brand and products. Blocking them is reasonable if your content is your product, such as paid research or a news archive. Many brands that sell software or services allow them, because being known to the model is the goal.

This template allows AI search and user agents, allows training bots, and blocks one training bot as an example of a selective opt-out. Replace the paths with your own private areas.

# Search and user agents: allow, so AI engines can cite you
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
Disallow: /admin/

# Training: allow (remove a line to opt that bot out)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Allow: /
Disallow: /admin/

# Example of a selective opt-out
User-agent: Bytespider
Disallow: /

User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml

One detail catches many teams: a crawler obeys only the most specific group that names it. If you name GPTBot in its own group, it ignores your User-agent: * rules, so repeat your Disallow lines in every named group.

Usually because of a CDN or firewall setting. Several CDN and security providers offer one-click "block AI bots" options or bot-fight modes that return an error to AI crawlers before robots.txt is ever read. Check:

1. AI bot policies in your CDN dashboard. Cloudflare, for example, now sorts bots into Search, Agent and Training and can block each class on all pages or only on pages with ads; blocking Training there also blocks multi-purpose crawlers such as Googlebot.
2. Bot protection rules that challenge "automated" traffic; verified AI crawlers should be exempt.
3. Rate limits and geo-blocking that hit crawler IP ranges.
4. Login walls or cookie walls that hide the main content.

Then test: fetch your key pages with each user-agent and confirm a 200 response with the full text in the HTML. Content that appears only after JavaScript runs may be invisible to some AI crawlers.

Yes. Anyone can fake a user-agent string, so the major vendors publish IP ranges for their crawlers; OpenAI, Perplexity and Google document theirs on the pages linked below. Good CDNs check these automatically and label the traffic as a verified bot. Block fake bots, not real ones.

Look at three places: robots.txt (are the bots allowed?), your CDN's bot logs (are they getting 200 or 403?) and your server logs (which bots visit, and how often?). Then check the outcome: run your key prompts and see whether engines cite your pages. Ossvy's actionable insights flag pages that engines don't cite, and the free AI visibility check gives a first snapshot.

For the bigger picture of how crawler access fits into AI visibility, read our GEO guide and how ChatGPT recommends brands.

Allow the AI search crawlers and user agents (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User) so engines can cite you. Decide on training bots (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) as a separate business choice. Then check that your CDN agrees with your robots.txt.