AI crawlers

What are the main AI crawlers?#

The main AI crawlers belong to OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User), Anthropic (ClaudeBot, Claude-SearchBot, Claude-User) and Perplexity (PerplexityBot, Perplexity-User), plus Google, whose AI features ride on Googlebot with a separate Google-Extended token for Gemini.

Each vendor splits its bots by job, and each job has its own robots.txt token, so you can make a separate decision for each. These are the bots that decide whether you appear in the major AI answer engines, as each vendor documents them in September 2026:

Two things no table of user agents can show. Brave Search, the index behind Claude’s web search, has no user agent of its own and won’t crawl what Googlebot is blocked from. And agentic browsers such as Perplexity’s Comet and ChatGPT’s agent arrive with ordinary Chrome user agents, so they look like people.

Should I block AI crawlers?#

Blocking AI crawlers makes sense only for the training bots, and only if keeping your content out of future models matters more to you than being in them; the search and user-triggered bots are the ones that get you cited, so block those and you drop out of that engine’s answers.

The training decision is a real trade-off, not a free win. ChatGPT searched the web on about a third of prompts in early 2026, per Semrush’s clickstream data; the rest were answered from what the model learned in training. Opting out protects your content from reuse, and it also means the next model knows less about your brand and your category. See LLM training data.

How do I check which AI crawlers visit my site?#

To see which AI crawlers visit your site, search your server or CDN access logs for each bot’s user-agent token, then verify the source IPs against the vendor’s published list, because user agents are trivial to fake.

# Hits per AI bot in this log
grep -oE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|CCBot" access.log \
  | sort | uniq -c | sort -rn

Hits from the user-triggered bots are the most useful signal: each ChatGPT-User, Claude-User or Perplexity-User request is a live conversation that needed your page, and Cloudflare saw this kind of user-action crawling grow more than 15-fold across its network in 2025. Watch which URLs they fetch most, because those are the pages real questions are pulling into answers. Two gaps remain. Google-Extended never appears in logs because it isn’t a crawler, and Comet sends a standard Chrome user agent from the user’s own IP. Logs show who read you; to see who cited you, you need prompt tracking.

AI crawlers vs search engine crawlers: what's the difference?#

Search engine crawlers such as Googlebot and Bingbot index pages to rank them as links, while AI crawlers fetch pages to train models, feed an AI answer index or read a page live for one user — and unlike Googlebot, the major AI crawlers don’t run JavaScript.

The last row is why the distinction matters. Cloudflare’s 2025 Year in Review found that crawling for model training made up the overwhelming majority of AI crawler traffic, and measured crawl-to-referral ratios as high as 500,000 to 1 for Anthropic and 3,700 to 1 for OpenAI, with Perplexity lowest of the major platforms. Training crawls send no visitors by design; search-type AI bots can. Because AI crawlers read raw HTML, server-side rendering matters more for them than for Google.

Common mistakes with AI crawlers#

The most common AI crawler mistakes are blocking a search bot when you meant to block training, trusting robots.txt while a CDN setting blocks the bots anyway, and assuming an AI-bot block can’t touch Googlebot.

Related terms#

• AI crawlersrobots.txtA plain-text file at a site’s root that tells crawlers which paths they may fetch, rule by rule for each user agent — a voluntary standard that reputable search and AI bots follow, which controls crawling but not whether a URL gets indexed.Read the entry
• AI crawlersGPTBotOpenAI’s web crawler for collecting content that may be used to train its AI models; blocking it in robots.txt opts a site out of training but does not remove it from ChatGPT search, which uses a separate crawler, OAI-SearchBot.Read the entry
• AI crawlersOAI-SearchBotOpenAI’s search crawler, the bot that surfaces websites in ChatGPT search answers, which makes it the OpenAI user agent to allow for ChatGPT visibility — unlike GPTBot, which collects content only for model training.Read the entry
• AI crawlersClaudeBotAnthropic’s crawler for collecting web content that may be used to train Claude models, one of three Anthropic bots alongside Claude-SearchBot, which indexes pages for Claude’s search results, and Claude-User, which fetches pages when a user asks.Read the entry
• AI crawlersPerplexityBotPerplexity’s crawler for indexing web pages so they can be surfaced and cited in Perplexity’s answers — Perplexity says it is not used to train foundation models — while a separate agent, Perplexity-User, fetches pages live when a user asks.Read the entry
• AI crawlersGoogle-ExtendedA robots.txt product token, not a separate crawler, that lets site owners stop Google from using their content to train Gemini models and to ground answers in the Gemini app, without affecting Google Search — including AI Overviews and AI Mode.Read the entry