PerplexityBot

What does PerplexityBot do?#

PerplexityBot crawls web pages to surface and link them in Perplexity’s search results, feeding Perplexity’s own index of more than 200 billion URLs; Perplexity says it is not used to crawl content for training AI foundation models.

Perplexity’s bot documentation describes PerplexityBot as “designed to surface and link websites in search results on Perplexity” and says it is “not used to crawl content for AI foundation models.” Its crawls feed the index Perplexity built after leaving third-party search APIs, which, per its architecture write-up, “tracks over 200 billion unique URLs” and splits each page into passages that are retrieved and ranked individually. Its user agent:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)

• Honors robots.txt: yes, with changes taking up to 24 hours to apply.
• IP ranges: published at perplexity.com/perplexitybot.json.
• No JavaScript: Vercel’s crawler study found none of the major AI crawlers render it, so client-rendered text is invisible. See server-side rendering.
• Reach beyond perplexity.ai: the same index powers Perplexity’s APIs and its Comet browser.

PerplexityBot vs Perplexity-User: what's the difference?#

PerplexityBot crawls ahead of time to build Perplexity’s search index and honors robots.txt, while Perplexity-User fetches a page live when a user’s question needs it and, by Perplexity’s own account, generally ignores robots.txt because a person initiated the request.

The split matters if you ever want to limit Perplexity. A robots.txt rule stops the indexing crawler but not the live fetcher; a hard block needs a firewall rule on Perplexity-User‘s user agent and IP ranges. Comet, Perplexity’s browser, isn’t a bot at all: it sends a standard Chrome user agent from the user’s own IP and can’t be told apart in logs.

There’s history here too. In August 2025 Cloudflare reported Perplexity fetching pages with undeclared user agents when its declared bots were blocked; Perplexity disputed the account. The practical lesson either way: set your Perplexity policy explicitly rather than trusting a CDN default to get it right.

Should I block PerplexityBot?#

Blocking PerplexityBot rarely makes sense for a business that wants to be found: the bot isn’t used for model training, so a block buys no training opt-out, and it strips your full text from the index Perplexity’s answers cite — though your domain, headline and a short summary may still appear.

Perplexity’s help center says that if a page disallows PerplexityBot, “we may still index the domain, headline, and a brief factual summary.” Blocking doesn’t make you invisible; it makes you a thinner, less citable entry. The remaining reason to block is server load, and PerplexityBot honors robots.txt rate limits and backs off when a site struggles, so throttling usually solves that without leaving the index.

# Eligible for Perplexity search and live fetches
User-agent: PerplexityBot
Allow: /
Disallow: /account/
Disallow: /cart/
 
User-agent: Perplexity-User
Allow: /
 
Sitemap: https://example.com/sitemap.xml

Perplexity is also the AI engine where classic SEO carries over best: 28.6% of its citations ranked in Google’s top 10 in Ahrefs’ 2025 study, the highest of any assistant. A block throws that overlap away. See the Perplexity SEO guide for what its ranking rewards.

How do I check whether PerplexityBot visits my site?#

Check whether PerplexityBot visits your site by searching your access logs for PerplexityBot and Perplexity-User, then verifying the IPs against perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json, since both user agents are easy to spoof.

grep -oE "PerplexityBot|Perplexity-User" access.log | sort | uniq -c
 
# Pages Perplexity fetched live for users
grep "Perplexity-User" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Perplexity-User hits are the most telling: each one is a live question that needed your page, so the URLs it fetches most are the pages Perplexity is pulling into answers right now. Give any new rule a day before judging it, since Perplexity says robots.txt changes can take up to 24 hours. Requests claiming to be PerplexityBot from IPs outside the published list aren’t Perplexity’s crawler, and blocking them at the firewall costs you nothing. Compare verified Perplexity-User hits with visits from perplexity.ai in analytics to see how often a fetch turns into a click. Perplexity adds no UTM tags, so its traffic usually lands in Referral unless you give it a custom channel. See AI referral traffic, and for citations themselves, prompt tracking.

Related terms#

• AI crawlersAI crawlersAutomated bots run by AI companies that fetch web pages for one of three jobs — training models, building an AI search index, or retrieving a page live for a user’s question — and because each job uses its own user agent, each can be allowed or blocked separately.Read the entry
• AI crawlersrobots.txtA plain-text file at a site’s root that tells crawlers which paths they may fetch, rule by rule for each user agent — a voluntary standard that reputable search and AI bots follow, which controls crawling but not whether a URL gets indexed.Read the entry
• MeasurementAI referral trafficThe visits a website receives from links inside AI assistants and answer engines such as ChatGPT, Perplexity, Gemini, Claude and Copilot, identified in analytics by referrer domains like chatgpt.com and perplexity.ai.Read the entry
• SSRTechnical SEOServer-side renderingThe practice of generating a page’s full HTML on the server before sending it, so crawlers that don’t run JavaScript — which includes most AI crawlers — can read the content in the first response.Read the entry
• AI crawlersClaudeBotAnthropic’s crawler for collecting web content that may be used to train Claude models, one of three Anthropic bots alongside Claude-SearchBot, which indexes pages for Claude’s search results, and Claude-User, which fetches pages when a user asks.Read the entry
• AI crawlersOAI-SearchBotOpenAI’s search crawler, the bot that surfaces websites in ChatGPT search answers, which makes it the OpenAI user agent to allow for ChatGPT visibility — unlike GPTBot, which collects content only for model training.Read the entry