Claude SEO: how to get found, read and quoted by Claude

Claude answers from what it learned in training unless the question needs something current. Anthropic’s web search documentation lists what triggers a search: recent events, current prices and statistics, organizations, people or products that might have changed, and explicit requests to look something up. Settled facts, maths, coding concepts and creative work are answered directly. In Profound’s July 2026 test, Claude searched on 36.6% of prompts, and words like “best”, “near me” or a year made a search more likely.

In the Claude apps that decision is increasingly Claude’s alone. Since 16 September 2026 Anthropic has been folding Cowork into every conversation on Pro and Max, and its web search help article says that on this new experience “there’s no web search toggle. Claude searches the web when it helps.”

Three vendors carry the label “Web Search”

Anthropic’s product docs never name a search provider. Its subprocessor list does, and as of 26 October 2026 three companies sit under the purpose “Web Search”:

Anthropic hasn’t said which vendor serves which query or how results are combined. The public record is the timeline. Simon Willison spotted Brave on the list in March 2025, found a BraveSearchParams schema in Claude’s search tool, and saw Claude’s ten citations for a test query exactly match Brave’s results. The Trust Center’s update log describes TurboPuffer (6 May 2026) and Parallel (14 October 2026) in the same words: each “provides web search for all Anthropic products except Claude for Government.”

One search turn, step by step

1. 1

   Claude decides a search would help

   Current, changing or specific questions trigger one; settled knowledge doesn’t. Developers building on Claude can cap searches per request with max_uses.

   Your move: Cover the things that make Claude search, such as prices, specs, comparisons and recent changes, not only evergreen explainers.

2. 2

   It writes its own queries

   Anthropic says simple questions take one to three searches and comparisons ten or more. In Profound’s data, 94% of Claude’s queries carried a year.

   Your move: Use an honest year in titles and headings wherever freshness matters.

3. 3

   Results come back with a page age

   Each result carries a URL, a title and page_age, which Anthropic describes as when the site was last updated. Newer tool versions let Claude run code that filters results before it reads them.

   Your move: Keep a visible updated date and an accurate dateModified, so the age Claude sees is the real one.

4. 4

   Claude fetches the pages it needs

   Full pages arrive through web fetch as text, HTML or PDF. No JavaScript runs, and Claude can only fetch URLs that already appeared in the conversation or its search results.

   Your move: Put the answer in the server-rendered HTML, near the top.

5. 5

   It answers with citations attached

   Citations are always on for web search. Each one carries the source URL, its title and up to 150 characters of cited text.

   Your move: State key facts in sentences that still make sense when lifted out alone.

Anthropic’s crawler article names three user agents, each tied to a different outcome. Block the wrong one and you lose something you meant to keep. Flip them below to see the robots.txt each choice produces.

User-agent: Claude-SearchBotAllow: /User-agent: Claude-UserAllow: /User-agent: ClaudeBotAllow: /

Two rules hold whatever you choose. A group that names a bot overrides User-agent: *, so a blanket block stops applying once the bot is named. And Anthropic asks for the rules on every subdomain you want covered, since each serves its own robots.txt.

Older templates still list anthropic-ai and Claude-Web. Anthropic’s current article names neither, so rules for them don’t govern the three bots above. Make sure your file names the current tokens.

Allowing Anthropic’s bots gets you read. Getting found also depends on the vendors’ own indexes, and two of the three run crawlers you can influence.

Brave Search

Brave’s crawler page says its bot “does not advertise a differentiated user agent”, and that “if a domain or page is not crawlable by Googlebot, then Brave Search’s bot will not crawl it either.” There’s no Brave line to add to robots.txt, so a rule or WAF setting that blocks Googlebot quietly costs you Brave as well. Brave also notes that robots.txt doesn’t stop indexing: use noindex and ask for a re-fetch.

Brave finds new pages partly through its opt-in Web Discovery Project, where Brave browser users report pages anonymously. MERJ’s July 2026 teardown of the reporting client turns its rules into a hygiene list:

• Rank somewhere else first. One opted-in user who sees your URL in another engine’s results can report it. A page found only by browsing needs reports from 20 different users on different networks.
• Keep URLs clean. Without a qualifying canonical, a second query parameter or a query string over 22 characters marks the URL private.
• Put canonical and noindex in the HTML. The client reads <link rel="canonical"> and noindex from the page itself, never from the Link or X-Robots-Tag headers.
• Skip the redirects. Any redirect counts as private. The page must answer 200 at exactly the URL reported.
• Stay light. The check gives up after 10 seconds and refuses responses over 2 MB.
• Request a re-crawl after big changes. Brave’s submission form asks for a re-fetch. It doesn’t guarantee indexing.

Parallel Web Systems

Parallel says it runs “our own web-scale index with billions of pages.” Its crawler docs name ShapBot, which “helps discover and index websites for Parallel’s web APIs”, and its bot overview warns that restricting it “may reduce your site’s visibility and discoverability.” A separate Shap-User agent fetches pages on behalf of users. Anthropic hasn’t said what it uses Parallel for, but with Parallel on the list as a web-search vendor, allowing ShapBot is a cheap hedge.

TurboPuffer

TurboPuffer describes itself as “vector and full-text search built on object storage” and lists Anthropic as a customer. It’s a database, not a crawler: there’s nothing to allow or block, and nothing public about what it stores for Claude.

# Brave has no user agent of its own: it follows Googlebot's rulesUser-agent: GooglebotAllow: /# Parallel Web Systems, one of Claude's listed web-search vendorsUser-agent: ShapBotAllow: /

Then check where you stand. Search your target questions on search.brave.com with the year added, the way Claude writes them. A top-10 Brave result is the nearest thing to a public test of whether Claude can find you.

When Claude reads a page it uses web fetch, and Anthropic’s fetch documentation is plain about it: the tool “currently does not support websites dynamically rendered with JavaScript.” It accepts text, HTML and PDF, refuses anything else, and hands Claude the page’s text. Vercel’s crawler study saw the same on the crawling side: ClaudeBot downloaded JavaScript files, 23.84% of its requests, without executing them.

The answer in the HTML responseRequired

Server-side rendering, static generation or pre-rendering. A client-rendered app shows Claude an empty shell.

Robots.txt that allows Claude-UserRequired

Web fetch refuses URLs that robots.txt blocks, returning a url_not_allowed error, and Claude carries on without you.

No wall in front of the answerRequired

Anthropic’s bots don’t try to bypass CAPTCHAs, and a login or interstitial ends the fetch at the door.

The answer near the topHelps

Anthropic sizes an average 10 kB page at about 2,500 tokens. Developers cap fetches with max_content_tokens, and anything past the cap is truncated.

Clear headings and real tablesHelps

Newer fetch versions let Claude run code to filter a page before reading it. Our read: descriptive headings and HTML tables make the relevant part easier to keep.

Markdown on requestHelps

Cloudflare reports that Claude Code sends Accept headers asking for markdown. Serving it, as Cloudflare’s Markdown for Agents does, gives coding agents a cleaner read.

Structured dataUnconfirmed

Anthropic has said nothing about schema. Fetch returns page text, so facts that live only in JSON-LD may never reach the model. Repeat them in visible copy.

llms.txtUnconfirmed

Anthropic hasn’t said Claude reads other sites’ llms.txt. Ahrefs found 97% of the files it tracked got no requests in May 2026. Among the requests that did come in, Claude Code out-fetched every AI search bot and assistant.

IndexNowNo effect

Brave isn’t an IndexNow participant; the protocol’s list is Amazon, Bing, Naver, Seznam, Yandex and Yep. It still helps you with Bing.

Fetch results are also cached, so Anthropic warns the copy Claude reads “might not always reflect the latest version.” The quickest test is to request a page the way Claude does and look for the sentence you want quoted. If it isn’t there, Claude can’t see it either:

# Does your server (and WAF) give Claude the real content?curl -s -A "Claude-User" https://yoursite.com/pricing \  | grep -c "Plans start at"# 0 means the text is rendered by JavaScript, or the request was blocked

Web fetch is Claude’s default reader, not its only one. Three surfaces go further, and they change what “visible to Claude” means.

• Claude in Chrome is a browser extension on paid plans (Pro, Max, Team and Enterprise). Anthropic’s setup guide says it lets Claude “read, click, and navigate websites alongside you”, typing, filling in forms and running scheduled browser tasks. It works inside the user’s own Chrome, so pages render as they would for that user.
• Research “runs multiple searches that build on each other and decides what to look into next”, according to Anthropic’s Research guide. Its engineering write-up says complex questions can use more than 10 subagents, and that early versions “consistently chose SEO-optimized content farms over authoritative but less highly-ranked sources” until the evaluation asked for “primary sources over lower-quality secondary sources.”
• Cowork in every conversation. Anthropic’s release notes date the merge of chat and Cowork, its mode for longer multi-step tasks, to 16 September 2026 for Pro and Max. Agent-style work increasingly starts in an ordinary chat.

For an agent, a page has to work as well as read well. Our read: the pages an agent is sent to act on, like pricing, sign-up, booking and comparisons, should load without a CAPTCHA wall, keep key facts in text rather than images, and use buttons and form fields with clear labels.

Claude’s citations are built into the search tool, not added afterwards. Anthropic’s API docs say citations “are always enabled for web search”, and each carries the source’s URL, its title and cited_text: “up to 150 characters of the cited content.” In the apps, Anthropic’s help article promises that “every response includes citations”, with direct citations to sources, links for further reading and relevant quotes.

That 150-character window is the unit you’re competing for. A sentence that names its subject and carries its own number can be quoted whole. One that leans on the paragraph before it gets cut mid-thought, or skipped for a page that says it more plainly.

• One fact per sentence, subject named. Use the product or company name instead of “it” or “we” in the sentences you most want quoted.
• Answer first. Put the direct answer straight under each heading, then the context.
• Numbers with units and dates. “$8 per user per month” beats “affordable”, and “as of October 2026” beats “recently”.
• Comparisons in real HTML tables. Specs and prices in a table are text Claude can read. The same numbers in an image aren’t.
• An honest updated date. Claude’s queries usually carry a year and every result arrives with a page_age, so a visible, accurate date is part of the answer.

No study sees inside Claude’s selection, and they don’t all agree. These are the patterns with the most support, each marked with how strong the evidence is.

• Pages that rank in Brave

  Observed in studies

  79.2% of cited URLs sat in Brave’s top 10 for the query in Profound’s July 2026 study, measured before Parallel joined the vendor list.

• Your own domain

  Observed in studies

  Brand and company sites took 64% of 379,321 Claude citations in Otterly’s June 2026 study of SaaS and tech queries. Pricing, docs and spec pages are citable assets.

• Primary sources

  Anthropic says so

  Anthropic’s Research evaluation rewards primary sources over lower-quality secondary ones. Original data and first-hand documentation are what it’s tuned to find.

• Fresh lists and comparisons

  Observed in studies

  Listicles were 36.4% of Claude’s classified citations against 19.7% for ChatGPT, per Profound. Being named in current “best X” lists is a second way in.

• A separate race from ChatGPT

  Observed in studies

  Claude and ChatGPT shared 8% of cited domains in Profound’s data and 13% in Otterly’s. Winning one says little about the other.

• Forums: the studies disagree

  Observed in studies

  Profound put forums and UGC at 0.9% of Claude’s citations and Otterly counted no Reddit citations in SaaS and tech, but Attrifast’s 12-vertical panel had Reddit at 7.9%. Check your own category first.

• Volatile from run to run

  Observed in studies

  Attrifast found a median of 3.6 domains per Claude answer, and only 33% of citations appeared in all three runs of the same prompt. Judge visibility over repeated runs.

• Placement can’t be bought

  Anthropic says so

  Anthropic’s February 2026 pledge says Claude will stay ad-free, with no sponsored links and no advertiser influence on its responses.

When a click from Claude carries a referrer, it’s claude.ai, with no UTM tags. Analytics vendors report that many clicks carry none, app traffic especially, and those land in Direct. Treat your Referral count as a floor.

GA4 added a default AI Assistant channel on 13 May 2026. Google’s launch note names Claude, but the channel definition lists “ChatGPT, Gemini, Deepseek, Copilot, or Grok.” Check where your claude.ai sessions actually land. If they’re still in Referral, add a custom channel group above Referral with a session-source rule:

# Session source matches regex: Claude only^(.*\.)?claude\.ai$# Every major assistant in one channel(chatgpt\.com|claude\.ai|perplexity\.ai|gemini\.google\.com|copilot\.microsoft\.com)

Your server logs show the reading side, and each user agent answers a different question:

# Anthropic and Parallel bot hits, by user agentgrep -oE "ClaudeBot|Claude-SearchBot|Claude-User|ShapBot" access.log | sort | uniq -c# The pages Claude fetched for users: your live-demand listgrep "Claude-User" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Verify crawler hits against Anthropic’s published IP ranges, and expect one exception: Claude Code’s WebFetch runs wherever Claude Code runs, often a developer’s own machine, so those Claude-User requests won’t come from Anthropic’s addresses. Rankbox’s free AI crawler log analyzer groups these hits by bot and page.

• Blocking ClaudeBot hides you from Claude.

  It only opts your future content out of training. Search indexing is Claude-SearchBot and live reading is Claude-User, each with its own rule.

• Claude searches Google, so Google rankings are enough.

  Anthropic’s listed web-search vendors are Brave, TurboPuffer and Parallel. Google still matters indirectly: Brave won’t crawl what Googlebot can’t, and a Google ranking helps Brave discover the URL.

• Claude will render my JavaScript app.

  Its fetch tool doesn’t run JavaScript. Only browser surfaces like Claude in Chrome render pages, and they act for one user at a time, not for search.

• An llms.txt file gets you cited.

  Anthropic has never said Claude reads other sites’ llms.txt. It can help coding agents that read developer docs. There’s no evidence it’s a search signal.

• Placement in Claude’s answers can be bought.

  Anthropic’s February 2026 pledge rules out sponsored links and advertiser influence on Claude’s responses.