OAI-SearchBot

What does OAI-SearchBot do?#

OAI-SearchBot crawls web pages so OpenAI can surface them in ChatGPT’s search features, which makes it the crawler that decides whether ChatGPT search answers can show and cite your site.

OpenAI’s crawler documentation says OAI-SearchBot is used to “surface websites in search results in ChatGPT’s search features,” and spells out what blocking it costs: “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.” OpenAI recommends allowing both the bot and requests from its published IP ranges. Its current user agent:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

Match on the OAI-SearchBot token rather than the full string, which changes between versions. ChatGPT search also draws on third-party providers — OpenAI names Microsoft — so being indexed by Bing matters too. But OAI-SearchBot feeds OpenAI’s own index, and allowing it is the only inclusion rule OpenAI documents. The ChatGPT SEO guide covers the rest of the pipeline.

OAI-SearchBot vs GPTBot: what's the difference?#

OAI-SearchBot crawls for ChatGPT search, so blocking it removes you from ChatGPT’s cited answers, while GPTBot crawls for model training, so blocking it only opts you out of training; OpenAI treats the two as independent settings.

Confusing the two is the most expensive robots.txt mistake in AI search: a site that blocks OAI-SearchBot to avoid training disappears from ChatGPT’s answers and gains nothing, because training runs on a different bot. If training is your concern, the bot to block is GPTBot. When both are allowed, OpenAI says it may use one crawl for both jobs, so allowing both doesn’t double your server load.

How do I allow OAI-SearchBot?#

Allow OAI-SearchBot by giving it its own Allow: / group in robots.txt, then making sure your CDN, firewall and bot-protection settings let its published IP ranges through — robots.txt alone isn’t enough if a security layer blocks the request first.

# ChatGPT search: required to be cited
User-agent: OAI-SearchBot
Allow: /
Disallow: /account/
Disallow: /checkout/

1. Check for blanket blocks. A User-agent: * group with Disallow: / shuts OAI-SearchBot out unless it has a group of its own.
2. Copy your private paths. A named group ignores the * group entirely, so repeat any Disallow lines you still need, as above.
3. Open the CDN. Allowlist the IPs in openai.com/searchbot.json and don’t serve them challenges. From 15 September 2026, new Cloudflare domains block Training and Agent bots by default on pages with ads while allowing Search bots — check what your own plan and settings actually do.
4. Wait about 24 hours. That’s how long OpenAI says robots.txt changes take to reach its systems.
5. Serve the answer in HTML. OpenAI documents no JavaScript rendering, and Vercel’s crawler study saw OpenAI’s crawler fetch scripts without running them. See server-side rendering.

How do I check whether OAI-SearchBot visits my site?#

Check whether OAI-SearchBot visits your site by searching your access logs for its token, verifying the IPs against openai.com/searchbot.json, and requesting a key page with its user agent to confirm your server returns the real content rather than a challenge.

# OpenAI bot hits by type
grep -oE "OAI-SearchBot|ChatGPT-User|GPTBot|OAI-AdsBot" access.log | sort | uniq -c
 
# Does the bot get your real content?
curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" \
  https://yoursite.com/pricing | grep -c "Plans start at"
 
# 0 = the text is rendered by JavaScript, or a WAF is serving a challenge

No hits at all usually means one of three things: the bot is blocked (look for 403s in your CDN’s firewall log), the site is new and hasn’t been discovered, or requests are dropped before they reach your logs. Remember that a curl from your own machine only tests user-agent rules; an IP-based bot rule can still treat the real bot differently. Logs prove access, not citations. To see whether ChatGPT actually cites you, run a fixed set of buyer prompts on a schedule, which is prompt tracking, and watch AI referral traffic from chatgpt.com.

Common mistakes with OAI-SearchBot#

The most common OAI-SearchBot mistakes are blocking it while trying to block training, leaving a CDN or bot-fight setting that challenges it, and using a robots.txt Disallow to hide a page that needs a noindex instead.

Related terms#

• AI crawlersGPTBotOpenAI’s web crawler for collecting content that may be used to train its AI models; blocking it in robots.txt opts a site out of training but does not remove it from ChatGPT search, which uses a separate crawler, OAI-SearchBot.Read the entry
• AI crawlersAI crawlersAutomated bots run by AI companies that fetch web pages for one of three jobs — training models, building an AI search index, or retrieving a page live for a user’s question — and because each job uses its own user agent, each can be allowed or blocked separately.Read the entry
• AI crawlersrobots.txtA plain-text file at a site’s root that tells crawlers which paths they may fetch, rule by rule for each user agent — a voluntary standard that reputable search and AI bots follow, which controls crawling but not whether a URL gets indexed.Read the entry
• SSRTechnical SEOServer-side renderingThe practice of generating a page’s full HTML on the server before sending it, so crawlers that don’t run JavaScript — which includes most AI crawlers — can read the content in the first response.Read the entry
• MeasurementAI referral trafficThe visits a website receives from links inside AI assistants and answer engines such as ChatGPT, Perplexity, Gemini, Claude and Copilot, identified in analytics by referrer domains like chatgpt.com and perplexity.ai.Read the entry
• Technical SEOIndexingThe step in which a search engine processes a crawled page and stores it in its searchable database; only indexed pages can rank, or be retrieved for AI answers built on that index, and being crawled does not guarantee being indexed.Read the entry