robots.txt

How does robots.txt work?#

robots.txt works by listing groups of rules, each starting with one or more User-agent lines, that tell a named crawler which URL paths it may or may not fetch; a crawler reads the file at the site’s root before crawling and follows the single group that best matches its name.

The format was first proposed by Martijn Koster in 1994 and became an internet standard, RFC 9309, in September 2022. The rules that matter most in practice:

• One file per host. It must sit at /robots.txt and covers only that protocol, host and port, so blog.example.com needs its own.
• The most specific group wins. A crawler follows the group that names it and ignores User-agent: * entirely. Google never merges a named group with the * group.
• The longest matching path wins within a group. When an Allow and a Disallow match equally, the RFC and Google both favor the Allow.
• Matching rules: user-agent names are case-insensitive, paths are case-sensitive, * matches any run of characters and $ anchors the end of a URL.
• It’s cached. The RFC says crawlers shouldn’t rely on a cached copy for more than 24 hours; OpenAI and Perplexity say changes take up to about a day to apply.

One rule sits above the rest: robots.txt is a request, not a lock. The RFC says its rules “are not a form of access authorization,” and Google notes that while reputable crawlers obey them, “other crawlers might not.” See AI crawlers for which AI bots honor it.

How do I set up robots.txt for AI crawlers?#

Set up robots.txt for AI crawlers by giving each search and user-triggered bot an explicit Allow, making one deliberate decision for all the training tokens, and keeping Googlebot allowed, since Google’s AI features and Claude’s search index both depend on it.

# Search engines and AI search indexes: allow
User-agent: Googlebot
User-agent: Bingbot
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /account/
 
# Live fetches for a user's question: allow
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Allow: /
Disallow: /account/
 
# Model training: your call (change Allow to Disallow to opt out)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Allow: /
Disallow: /account/
 
# Everyone else
User-agent: *
Disallow: /account/
 
Sitemap: https://example.com/sitemap.xml

• One group can name several bots. RFC 9309 allows multiple User-agent lines above one set of rules, which keeps the file short.
• Every group repeats your private paths, because a named group doesn’t inherit anything from *.
• Disallowing Google-Extended also ends Gemini-app grounding, not just training. It’s the one training token with a search-side cost. See Google-Extended.
• ChatGPT-User and Perplexity-User may ignore robots.txt anyway, by their vendors’ own account. Use a firewall rule if you need a hard block.
• Then check the CDN. A bot-management rule can block a bot that robots.txt allows.

The AI robots.txt generator produces a starting file you can adapt.

robots.txt vs noindex: what's the difference?#

robots.txt controls whether a crawler may fetch a URL, while a noindex directive controls whether a fetched page may appear in results — so a URL blocked in robots.txt can still be indexed from links elsewhere, and the crawler must be allowed in to see a noindex at all.

Google says robots.txt “is not a mechanism for keeping a web page out of Google”: a disallowed URL “can still be indexed if linked to from other sites,” appearing with its address but no description. OpenAI says the same of ChatGPT, where a site opted out of OAI-SearchBot “can still appear as navigational links,” and Brave notes that “robots.txt is not used to prevent a page from being indexed.”

Listing a path in robots.txt also advertises it: the RFC warns that doing so “exposes them publicly.” Never use it to hide anything sensitive. See indexing and snippet controls.

How do I test robots.txt?#

Test robots.txt by fetching the live file from every host, checking which group each crawler you care about actually matches, and confirming in Search Console’s robots.txt report that Google fetched it without errors, then request a key page with each bot’s user agent to catch CDN blocks.

# 1. The file as crawlers see it (repeat for every host)
curl -s https://example.com/robots.txt
curl -s https://blog.example.com/robots.txt
 
# 2. The status code changes the meaning (see below)
curl -s -o /dev/null -w "%{http_code}\n" https://example.com/robots.txt
 
# 3. Can a bot actually get a page? Catches user-agent blocks at the CDN
curl -s -o /dev/null -w "%{http_code}\n" -A "Claude-SearchBot" https://example.com/pricing

• Status codes change everything. Under RFC 9309, a 4xx on robots.txt means crawlers may fetch anything, and a 5xx means they must assume everything is disallowed. Google stops crawling for 12 hours after a 5xx, then falls back to a cached copy for up to 30 days.
• Search Console’s robots.txt report lists the files Google found for your top hosts, when it last fetched them, and any parse errors. See Google Search Console.
• A curl with a bot’s user agent tests user-agent rules only. IP-based bot management can still treat the real bot differently, so check your CDN’s firewall log for challenges too.

Common mistakes with robots.txt#

The most common robots.txt mistakes are a leftover Disallow: / from staging, blocking a search bot instead of a training bot, forgetting that named groups ignore the * rules, and assuming the file you wrote is the file being served.

Related terms#

• AI crawlersAI crawlersAutomated bots run by AI companies that fetch web pages for one of three jobs — training models, building an AI search index, or retrieving a page live for a user’s question — and because each job uses its own user agent, each can be allowed or blocked separately.Read the entry
• Technical SEOIndexingThe step in which a search engine processes a crawled page and stores it in its searchable database; only indexed pages can rank, or be retrieved for AI answers built on that index, and being crawled does not guarantee being indexed.Read the entry
• Technical SEOSnippet controlsRobots directives and HTML attributes — nosnippet, max-snippet and data-nosnippet — that limit how much of a page search engines may quote in results, and in Google they also limit what AI Overviews and AI Mode can use.Read the entry
• Technical SEOXML sitemapA machine-readable file listing a site’s canonical URLs, optionally with last-modified dates, so search engines can discover pages and prioritize recrawling the ones that changed — a discovery aid, not a ranking factor or a guarantee of indexing.Read the entry
• AI crawlersGPTBotOpenAI’s web crawler for collecting content that may be used to train its AI models; blocking it in robots.txt opts a site out of training but does not remove it from ChatGPT search, which uses a separate crawler, OAI-SearchBot.Read the entry
• AI crawlersllms.txtA proposed standard for a Markdown file, usually at a site’s root, that gives large language models a curated map of a site’s key pages; unlike robots.txt it grants or blocks nothing, and no major AI search engine has confirmed using it.Read the entry