GPTBot

What is GPTBot used for?#

GPTBot is used to collect public web content that may be used to train OpenAI’s generative AI foundation models, and for nothing else: it doesn’t decide what appears in ChatGPT search, which is the job of a separate crawler, OAI-SearchBot.

OpenAI’s crawler documentation describes GPTBot as the bot that crawls “content that may be used in training our generative AI foundation models,” and gives the opt-out in one line: “Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models.” It identifies itself with this user agent:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

• Honors robots.txt: yes, and OpenAI says changes take about 24 hours to reach its systems.
• IP ranges: published at openai.com/gptbot.json, so you can tell real GPTBot traffic from imitators.
• Scale: GPTBot generated 7.5% of all verified bot traffic Cloudflare saw in 2025, more than Bingbot’s 6%.
• Forward-looking only: a block covers future crawls. OpenAI’s documentation doesn’t describe removing content collected before it.

Should I block GPTBot?#

Block GPTBot only if keeping your content out of future OpenAI models matters more to you than having those models know your brand; blocking it has no effect on ChatGPT search, so for a business that wants to be recommended, allowing it is usually the better trade.

The case for allowing rests on the answers that never search. Semrush’s clickstream data found ChatGPT searched the web on about a third of prompts in February 2026; the rest came from what the model learned in training, which is where LLM training data decides whether your brand comes up at all. A GPTBot block is also only partial: Common Crawl’s CCBot builds an open archive that has been a major source of LLM training data, so blocking GPTBot alone doesn’t opt you out of training in general.

GPTBot vs OAI-SearchBot: what's the difference?#

GPTBot collects content for training OpenAI’s models, while OAI-SearchBot crawls pages so they can be shown and cited in ChatGPT search — two independent robots.txt settings, so you can block training and still appear in search.

OpenAI puts the independence in writing: “a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot.” When both are allowed, OpenAI “may use the results from just one crawl for both use cases,” so allowing both doesn’t double your crawl load. A fourth bot, OAI-AdsBot, only checks landing pages submitted as ChatGPT ads. See AI crawlers for every vendor’s equivalents.

How do I block GPTBot?#

To block GPTBot, add a User-agent: GPTBot group with Disallow: / to the robots.txt of every host you want covered; OpenAI applies it within about 24 hours, and you can opt out only part of a site by disallowing specific paths instead.

# Opt out of OpenAI model training
User-agent: GPTBot
Disallow: /
 
# Stay in ChatGPT search
User-agent: OAI-SearchBot
Allow: /

To keep only premium sections out of training, disallow just those paths:

User-agent: GPTBot
Disallow: /research/
Disallow: /courses/

• Repeat it per host. robots.txt applies only to the host that serves it, so blog.example.com needs its own file.
• Named groups don’t inherit * rules. Once GPTBot has its own group, it ignores everything under User-agent: *, so copy across any paths you also need kept private.
• For a hard block, add a firewall rule on the ranges in openai.com/gptbot.json. GPTBot honors robots.txt, but a firewall rule doesn’t depend on any bot’s good behavior.
• Read the file as served before editing it. CDN features such as Cloudflare’s managed robots.txt can prepend AI training rules you never wrote, so your decision may already have been made for you.

How do I check whether GPTBot visits my site?#

To check whether GPTBot visits your site, search your access logs for the GPTBot user-agent token, then confirm the requesting IPs fall inside OpenAI’s published ranges at openai.com/gptbot.json, since anyone can copy the user agent.

# GPTBot hits by IP, then by URL
grep "GPTBot" access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head
grep "GPTBot" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head

Check the IPs against gptbot.json before drawing conclusions, because scrapers borrow well-known user agents. Expect some GPTBot requests for /robots.txt itself: that’s the bot rechecking your rules, not ignoring them. If you’ve blocked GPTBot and still see verified hits on disallowed pages after a day, check that the rule sits on the right host and that the token is spelled as a group of its own. GPTBot hits only tell you about training crawls. For ChatGPT search, look for OAI-SearchBot; for live conversations, ChatGPT-User — and for whether ChatGPT actually cites you, run a fixed prompt panel, which is prompt tracking.

Related terms#

• AI crawlersOAI-SearchBotOpenAI’s search crawler, the bot that surfaces websites in ChatGPT search answers, which makes it the OpenAI user agent to allow for ChatGPT visibility — unlike GPTBot, which collects content only for model training.Read the entry
• AI crawlersAI crawlersAutomated bots run by AI companies that fetch web pages for one of three jobs — training models, building an AI search index, or retrieving a page live for a user’s question — and because each job uses its own user agent, each can be allowed or blocked separately.Read the entry
• AI crawlersrobots.txtA plain-text file at a site’s root that tells crawlers which paths they may fetch, rule by rule for each user agent — a voluntary standard that reputable search and AI bots follow, which controls crawling but not whether a URL gets indexed.Read the entry
• How LLMs answerLLM training dataThe body of text — web crawls, books, code, licensed and forum content — that a language model learns from before release, and it shapes what the model says about a brand without searching, in a way that can’t be edited until the next model is trained.Read the entry
• AI crawlersClaudeBotAnthropic’s crawler for collecting web content that may be used to train Claude models, one of three Anthropic bots alongside Claude-SearchBot, which indexes pages for Claude’s search results, and Claude-User, which fetches pages when a user asks.Read the entry
• AI crawlersGoogle-ExtendedA robots.txt product token, not a separate crawler, that lets site owners stop Google from using their content to train Gemini models and to ground answers in the Gemini app, without affecting Google Search — including AI Overviews and AI Mode.Read the entry