---
title: "AI Crawlers List 2026: GPTBot, ClaudeBot, PerplexityBot"
description: "Every major AI crawler and what it does: search, user fetches or training. Plus a robots.txt template and the CDN settings that silently block AI bots."
canonical: "https://ossvy.com/blog/ai-crawlers-list"
lang: "en"
dir: "ltr"
dateModified: "2026-10-09"
type: "blog-post"
author: "Selin Arıkan"
reviewedBy: "Karim Nassar"
datePublished: "2026-10-09"
category: "Guides"
source: "Ossvy (https://ossvy.com)"
---

# The AI crawler list: GPTBot, ClaudeBot, PerplexityBot and how to allow them

Every major AI crawler, what it does, and a robots.txt that lets AI search engines cite you while you decide what to allow for training.

> In short: AI crawlers come in three kinds: search crawlers that build the index assistants cite (OAI-SearchBot, Claude-SearchBot, PerplexityBot), user agents that fetch a page on request (ChatGPT-User, Perplexity-User) and training bots (GPTBot, ClaudeBot, Google-Extended). Allow the first two to be cited.

- Blocking a search crawler removes you from that assistant's citations.
- Training and search are separate bots; you can decide on each.
- Check your CDN: bot protection can block AI crawlers despite robots.txt.

Breadcrumb: [Home](https://ossvy.com/index.md) › [Blog](https://ossvy.com/blog.md) › [Guides](https://ossvy.com/blog/category/guides.md) › The AI crawler list: GPTBot, ClaudeBot, PerplexityBot and how to allow them

## What are AI crawlers?

AI crawlers are bots that AI companies use to read web pages, either to answer users' questions with up-to-date sources or to collect training data for future models. They identify themselves with a user-agent name and, at the major vendors, follow the rules in your robots.txt file ([RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html)). Some vendors note that fetches triggered directly by a user may be handled differently, so read each vendor's page.

The important thing is that one company usually runs several bots with different jobs. Blocking "OpenAI" in one line can remove you from ChatGPT search, even if all you wanted was to opt out of training.

## Which AI crawlers should you know?

The table lists the user-agent tokens that the major vendors document. Each vendor's own page is linked under Sources.

| User-agent | Company | Job | Effect of blocking |
| --- | --- | --- | --- |
| `OAI-SearchBot` | OpenAI | Builds the index for ChatGPT search | Your pages aren't shown as ChatGPT search sources |
| `ChatGPT-User` | OpenAI | Fetches a page when a ChatGPT user or GPT asks for it | ChatGPT can't open your page on request |
| `GPTBot` | OpenAI | Collects data that may train future models | Future models learn less about you from your site |
| `Claude-SearchBot` | Anthropic | Indexes pages for Claude's search results | Fewer citations in Claude |
| `Claude-User` | Anthropic | Fetches pages for a Claude user's request | Claude can't open your page on request |
| `ClaudeBot` | Anthropic | Collects training data | Training opt-out |
| `PerplexityBot` | Perplexity | Builds Perplexity's search index | Your pages aren't shown as Perplexity sources |
| `Perplexity-User` | Perplexity | Fetches a page a user asks about | Perplexity can't open your page on request |
| `Googlebot` | Google | Search index, also used for AI Overviews and AI Mode | You disappear from Google Search, not only AI |
| `Google-Extended` | Google | A robots token, not a separate crawler: controls use in Gemini models and grounding | Opts out of Gemini use; doesn't affect Search or AI Overviews |
| `Bingbot` | Microsoft | Bing index, also used by Microsoft Copilot | You disappear from Bing and Copilot answers |
| `Applebot` | Apple | Search for Siri and Spotlight | Missing from Apple search features |
| `Applebot-Extended` | Apple | A token that controls training use of Applebot data | Training opt-out |
| `CCBot` | Common Crawl | Open web archive used by many model builders | Missing from Common Crawl datasets |
| `meta-externalagent` | Meta | Collects data for Meta's AI products and training | Training opt-out |
| `Bytespider` | ByteDance | Collects data for ByteDance's models | Training opt-out |

## Which AI crawlers should you allow?

Allow every search crawler and user agent if you want AI engines to recommend and cite you. These bots are how ChatGPT, Claude and Perplexity read your pages when they answer a question ([OpenAI](https://developers.openai.com/api/docs/bots), [Perplexity](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)). Blocking them doesn't stop the assistant from talking about you; it only stops it from reading your version.

Training crawlers are a business decision. Allowing them helps future models learn accurate facts about your brand and products. Blocking them is reasonable if your content is your product, such as paid research or a news archive. Many brands that sell software or services allow them, because being known to the model is the goal.

## What does a robots.txt for AI search look like?

This template allows AI search and user agents, allows training bots, and blocks one training bot as an example of a selective opt-out. Replace the paths with your own private areas.

```
# Search and user agents: allow, so AI engines can cite you
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
Disallow: /admin/

# Training: allow (remove a line to opt that bot out)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Allow: /
Disallow: /admin/

# Example of a selective opt-out
User-agent: Bytespider
Disallow: /

User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml
```

One detail catches many teams: a crawler obeys only the most specific group that names it. If you name `GPTBot` in its own group, it ignores your `User-agent: *` rules, so repeat your `Disallow` lines in every named group.

## Why are AI crawlers blocked even when robots.txt allows them?

Usually because of a CDN or firewall setting. Several CDN and security providers offer one-click "block AI bots" options or bot-fight modes that return an error to AI crawlers before robots.txt is ever read. Check:

1. **AI bot policies** in your CDN dashboard. Cloudflare, for example, now sorts bots into Search, Agent and Training and can block each class on all pages or only on pages with ads; blocking Training there also blocks multi-purpose crawlers such as Googlebot.
2. **Bot protection rules** that challenge "automated" traffic; verified AI crawlers should be exempt.
3. **Rate limits and geo-blocking** that hit crawler IP ranges.
4. **Login walls or cookie walls** that hide the main content.

Then test: fetch your key pages with each user-agent and confirm a 200 response with the full text in the HTML. Content that appears only after JavaScript runs may be invisible to some AI crawlers.

## Can you verify that a bot is really who it says it is?

Yes. Anyone can fake a user-agent string, so the major vendors publish IP ranges for their crawlers; OpenAI, Perplexity and Google document theirs on the pages linked below. Good CDNs check these automatically and label the traffic as a verified bot. Block fake bots, not real ones.

## How do you check whether AI crawlers can reach your site?

Look at three places: robots.txt (are the bots allowed?), your CDN's bot logs (are they getting 200 or 403?) and your server logs (which bots visit, and how often?). Then check the outcome: run your key prompts and see whether engines cite your pages. Ossvy's [actionable insights](https://ossvy.com/product/actionable-insights.md) flag pages that engines don't cite, and the [free AI visibility check](https://ossvy.com/tools/ai-visibility-check.md) gives a first snapshot.

For the bigger picture of how crawler access fits into AI visibility, read our [GEO guide](https://ossvy.com/blog/generative-engine-optimization.md) and [how ChatGPT recommends brands](https://ossvy.com/blog/how-chatgpt-recommends-brands.md).

## Summary

Allow the AI search crawlers and user agents (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User) so engines can cite you. Decide on training bots (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) as a separate business choice. Then check that your CDN agrees with your robots.txt.

## Sources

- [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) (2025, official)
- [Anthropic Help Center: Does Anthropic crawl data from the web?](https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (2025, official)
- [Perplexity: Perplexity crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) (2025, official)
- [Google Search Central: Google's common crawlers (Google-Extended)](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) (2025, official)
- [Apple: About Applebot](https://support.apple.com/en-us/119829) (2025, official)
- [Microsoft Bing Webmaster Tools: Which crawlers does Bing use?](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0) (2025, official)
- [Common Crawl: CCBot](https://commoncrawl.org/ccbot) (2025, official)
- [Meta for Developers: Meta web crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers) (2025, official)
- [Cloudflare changelog: Bots](https://developers.cloudflare.com/changelog/product/bots/) (2026, official)
- [IETF RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) (2022, standard)

*Bot names and purposes come from each vendor's public documentation, linked under Sources, checked on the publishing date. Vendors change their bots; we update this list when they do. Ossvy's own robots.txt follows the same policy.*

## Questions and answers

### Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot and ChatGPT-User, which you control separately. See how ChatGPT uses sources on the [ChatGPT visibility tracker](https://ossvy.com/product/ai-brand-visibility/chatgpt.md) page.

### Does blocking Google-Extended remove me from AI Overviews?

No. AI Overviews and AI Mode use Google's search index, built by Googlebot. Google-Extended only controls use in Gemini models and grounding. Track Google's AI answers on the [Gemini](https://ossvy.com/product/ai-brand-visibility/gemini.md) page.

### Do AI crawlers follow robots.txt?

The major vendors say their documented crawlers do, though some treat user-triggered fetches differently; check each vendor's page. Unknown scrapers often ignore it, which is why CDNs verify bots by IP. Check whether engines actually cite you with [citation tracking](https://ossvy.com/product/citation-tracking.md).

### Should I allow AI training crawlers?

If you want future models to describe your brand accurately, usually yes. If your content is what you sell, blocking can make sense. Either way, allow the search crawlers. Learn more in our [GEO guide](https://ossvy.com/blog/generative-engine-optimization.md).

### Can AI crawlers read JavaScript-rendered pages?

Not reliably. Put the main text, prices and FAQs in the server-rendered HTML. The [free AI visibility check](https://ossvy.com/tools/ai-visibility-check.md) shows whether engines mention you today.

Related: [Generative engine optimization (GEO): the complete guide for 2026](https://ossvy.com/blog/generative-engine-optimization.md) · [How ChatGPT decides which brands to recommend](https://ossvy.com/blog/how-chatgpt-recommends-brands.md) · [How to track ChatGPT and Perplexity traffic in GA4](https://ossvy.com/blog/ai-traffic-ga4.md)

**Find out what AI says about your brand**: Start with a free check, or let us walk you through your results in a 30-minute demo. [Start free](https://app.ossvy.com/signup) · [Book a demo](https://ossvy.com/contact?form=demo)

Source: https://ossvy.com/blog/ai-crawlers-list · Last updated: 2026-10-09