ClaudeBot

What is ClaudeBot used for?#

ClaudeBot is used to collect public web content that may contribute to training Anthropic’s Claude models; blocking it signals that your site’s future content should be excluded from training, and it does not remove you from Claude’s search results.

Anthropic’s help center says ClaudeBot “helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training.” Disallowing it “signals that the site’s future materials should be excluded from our AI model training datasets” — future, not past.

• Honors robots.txt: yes, including the non-standard Crawl-delay extension, and it won’t try to bypass CAPTCHAs.
• IP ranges: published at claude.com/crawling/bots.json, shared by all three Anthropic bots.
• Doesn’t run JavaScript: Vercel’s crawler study saw Anthropic’s crawler download script files without executing them, so client-rendered text never reaches it.
• Old tokens: anthropic-ai and Claude-Web aren’t in Anthropic’s current documentation. Leaving them in robots.txt is harmless but does nothing.

ClaudeBot is one of three bots with separate jobs, and the other two decide whether Claude can cite you. See AI crawlers for how every vendor splits the same three roles.

ClaudeBot vs Claude-SearchBot vs Claude-User: what's the difference?#

ClaudeBot collects training data, Claude-SearchBot indexes pages to improve Claude’s search results, and Claude-User fetches a page when a person’s conversation needs it — three separate robots.txt tokens, so you can opt out of training while staying fully visible in Claude’s answers.

Unusually, all three honor robots.txt, Claude-User included; OpenAI, Perplexity and Google all say their user-triggered fetchers may ignore it. So a Claude-User block really does work: Anthropic says it “prevents our system from retrieving your content in response to a user query.” That makes it easy to switch off Claude’s live reading by mistake with a broad rule.

Should I block ClaudeBot?#

Block ClaudeBot only if keeping your future content out of Claude’s training matters more than having future Claude models know your brand; ClaudeBot has no effect on Claude’s web search, so leave Claude-SearchBot and Claude-User allowed either way.

The trade-off is sharper for Claude than it looks. In Profound’s 2026 testing, Claude searched the web on 36.6% of prompts; the other 63.4% were answered from what the model already knew. Blocking ClaudeBot keeps your future content out of the model that answers most questions without searching. See LLM training data.

# Claude search index and live fetches: allow
User-agent: Claude-SearchBot
Allow: /
 
User-agent: Claude-User
Allow: /
 
# Model training: your call
User-agent: ClaudeBot
Disallow: /

Anthropic asks you to repeat the rules on every subdomain you want covered. If server load is the worry rather than training, slow ClaudeBot down instead of blocking it: Anthropic supports Crawl-delay, for example Crawl-delay: 1 under User-agent: ClaudeBot. And whatever you choose, make the same call for the other training tokens — GPTBot, Google-Extended, CCBot — or the opt-out is mostly symbolic.

How do I check whether ClaudeBot visits my site?#

Check whether ClaudeBot visits your site by searching your access logs for the ClaudeBot token and verifying the source IPs against claude.com/crawling/bots.json; count Claude-SearchBot and Claude-User separately, because they tell you about search, not training.

# Anthropic bot hits by type
grep -oE "ClaudeBot|Claude-SearchBot|Claude-User" access.log | sort | uniq -c
 
# Pages Claude fetched for users: your live-demand list
grep "Claude-User" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Requests for /robots.txt itself are the bots checking your rules, and Anthropic says all three follow them, so verified hits on disallowed paths a day after a change usually mean a typo in the token or a rule on the wrong host. Use the IP list to verify, and robots.txt to decide. Anthropic warns that blocking by IP “may not work correctly or persistently guarantee an opt-out,” because it stops the bot reading your robots.txt. Claude Code is the exception to IP checks, since its Claude-User requests come from each user’s own machine. And one crawler that matters for Claude won’t show up under any Anthropic name: Brave’s, which builds the index behind Claude’s web search and doesn’t advertise a user agent of its own.

Common mistakes with ClaudeBot#

The most common ClaudeBot mistakes are treating it as Claude’s search crawler, blocking Googlebot without realizing Claude’s search index follows Googlebot’s rules, and forgetting that robots.txt rules must be repeated on every subdomain.

Related terms#

• AI crawlersAI crawlersAutomated bots run by AI companies that fetch web pages for one of three jobs — training models, building an AI search index, or retrieving a page live for a user’s question — and because each job uses its own user agent, each can be allowed or blocked separately.Read the entry
• AI crawlersGPTBotOpenAI’s web crawler for collecting content that may be used to train its AI models; blocking it in robots.txt opts a site out of training but does not remove it from ChatGPT search, which uses a separate crawler, OAI-SearchBot.Read the entry
• AI crawlersrobots.txtA plain-text file at a site’s root that tells crawlers which paths they may fetch, rule by rule for each user agent — a voluntary standard that reputable search and AI bots follow, which controls crawling but not whether a URL gets indexed.Read the entry
• SSRTechnical SEOServer-side renderingThe practice of generating a page’s full HTML on the server before sending it, so crawlers that don’t run JavaScript — which includes most AI crawlers — can read the content in the first response.Read the entry
• How LLMs answerLLM training dataThe body of text — web crawls, books, code, licensed and forum content — that a language model learns from before release, and it shapes what the model says about a brand without searching, in a way that can’t be edited until the next model is trained.Read the entry
• AI crawlersPerplexityBotPerplexity’s crawler for indexing web pages so they can be surfaced and cited in Perplexity’s answers — Perplexity says it is not used to train foundation models — while a separate agent, Perplexity-User, fetches pages live when a user asks.Read the entry