How to Benchmark AI Search Performance

Start with access, because it sets the ceiling. A site that blocks an engine's search crawler can't win that engine with its own text. In Rankbox's AI Bot Crawler Census of 21,353 sites, 28.78% of top news sites fully block OAI-SearchBot, the crawler behind ChatGPT search, while only 1.28% of software and SaaS sites do. Before you compare yourself with a rival in ChatGPT, check that you are both in the race.

This post is about comparing AI search performance: which numbers to put side by side, how big a gap has to be before it's real, and how to read it. For collecting each number in depth, from logging answers to the control-prompt test, see our guide to measuring GEO.

What Benchmarking Adds to Measuring

Measuring tells you your numbers. Benchmarking AI search performance tells you whether they are good and whether they moved. That takes three comparisons, and each catches a mistake the others miss.

1. Against your baseline. Did your visibility rise since the last period, by more than the noise?
2. Against rivals. On the same prompts, in the same answers, who gets named more often?
3. Per engine. A blended score hides the engine where you are losing. The 2026 study "Don't Measure Once", which tested ChatGPT, Gemini, Google AI Mode and Perplexity, advises marketers to "set engine-specific visibility baselines rather than applying a single threshold across all AI search products." AI Mode concentrated citations most and Perplexity spread them most evenly.

As of September 2026, two vendors give you first-party numbers. Google's Search Generative AI performance report, live for all sites since 31 August 2026, counts how often your URLs appeared in AI Overviews and AI Mode. Microsoft's AI Performance report counts citations across Copilot, AI summaries in Bing and select partners, and its June 2026 update added two features built for benchmarking. Compare lets you "overlay a previous time period directly onto the current reporting view." Citation Share is "the percentage of citations attributed to your site out of all citations shown across all sites for that same grounding query." ChatGPT, Perplexity and Claude offer nothing similar, so for them your prompt panel is the benchmark.

The Five Numbers That Make Up AI Search Performance

The first three come from the same answers, so log every brand each answer names, not just yours. Your rivals' numbers come free. Our glossary entry on AI share of voice covers the two common ways to calculate the third.

AI referral sessions need a fix first. Google's default channel definitions describe the AI Assistant channel as traffic from "sources like ChatGPT, Gemini, Deepseek, Copilot, or Grok," excluding AI Overviews and AI Mode, so Perplexity and Claude go unnamed. Our GA4 guide to AI referral traffic has a custom channel that catches them. Set it up before your baseline.

Crawl coverage is the number most benchmarks skip. Pick 30 to 60 pages that should win answers, such as pricing, comparisons and core guides, and count how many each search crawler fetched in the last 30 days. The free AI crawler log analyzer lists the pages each bot fetched from a log export.

Check Crawl Access Before You Compare

A fair benchmark of AI search performance compares like with like. If a rival has shut out an engine's search crawler, its low score there reflects a choice, not weaker content. Here is what each vendor documents about blocking its search bot:

• OpenAI. Sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links," according to OpenAI's crawler overview.
• Perplexity. PerplexityBot "is designed to surface and link websites in search results on Perplexity," says Perplexity's crawler page, and it isn't used to train foundation models. Blocking it opts a site out of Perplexity search.
• Anthropic. Turning off Claude-SearchBot "may reduce your site's visibility and accuracy in user search results," per Anthropic's help center.

How to run the access check

1. Test your own robots.txt for OAI-SearchBot, PerplexityBot, Claude-SearchBot and Bingbot with the robots.txt tester.
2. Read each rival's robots.txt at /robots.txt and note, per engine, whether its search bot is allowed.
3. Mark each rival and engine pair as open or closed. Leave closed pairs out of the rival comparison for that engine.
4. Re-check every quarter. The census will re-run each quarter, and rivals change their files too.

The census shows how often a closed pair turns up. It's rare in B2B software and common in news, where 45.12% of top sites also block PerplexityBot. A GPTBot block alone usually leaves ChatGPT search open: across all 13,359 readable files in the census, only 8 sites block OAI-SearchBot while letting GPTBot in.

Robots.txt isn't the only gate. A quarter of e-commerce sites (25.1%) wouldn't serve robots.txt to the census crawler at all, a sign that firewalls decide at the edge. If your own crawl coverage drops to zero for one bot, check your CDN first, using our post on the Cloudflare challenge trap. After any access change, wait a day before you measure: OpenAI says robots.txt updates take about 24 hours to reach its systems, and Perplexity says up to 24 hours.

Size the Panel and Fix the Conditions

How many answers you need

One answer tells you almost nothing. When SparkToro had 600 volunteers run 12 prompts 2,961 times, lists of brands almost never repeated, and Rand Fishkin advises asking "usually at least 60-100X, then average these out." The "Don't Measure Once" authors recommend at least 7 runs per prompt per day, pooled over a rolling two-to-four-week window.

By hand, you pool instead. Use 30 to 50 unbranded buyer prompts, the same list for every brand, and add up the answers across the period. Then check whether a gap between two brands beats chance. The standard margin on a difference between two rates is:

margin = 1.96 × √( p1(1 − p1) / n1 + p2(1 − p2) / n2 )

Here is what it gives for two brands at 25% and 40% visibility:

Treat these as a floor. Repeated runs of one prompt aren't fully independent, so the true margin is wider. The same formula works for your own change over time: put last period's rate in as p1 and this period's as p2.

Freeze the conditions

AI answers change with who is asking, so benchmarking AI search performance needs a neutral asker. OpenAI says ChatGPT "may use relevant saved memories when rewriting a search query" and estimates your location from your IP address, which a VPN can change (OpenAI Help Center). A temporary chat can be set to Unpersonalized, so it skips saved memories and custom instructions (memory help page).

• Run every prompt logged out, or in an unpersonalized temporary session.
• Use the same location, or the same VPN exit, every time.
• Record the engine, mode and model shown, so a model change doesn't pass for a real gain.
• Change prompts in batches between periods, never mid-period.

Our post on how AI search uses user intent and context explains why these signals move answers.

Set a cadence

• Every week: run each prompt once in each engine. Spreading runs over weeks follows the rolling-window advice.
• Every month: pool the weeks into one benchmark period, update the sheet and export Bing's and Google's AI reports for the same dates.
• Every quarter: re-check rivals' robots.txt, retire dead prompts and add new ones in a batch.
• After a big change to your site's access or an engine's search mode: take a new baseline.

The Three-Way Benchmark Sheet: A Worked Example

Plannora is a made-up project management tool. It benchmarks itself against two made-up rivals, Loopcraft and Taskwell. Its panel is 40 unbranded prompts in ChatGPT, Perplexity and Google AI Mode, run once a week for three weeks a month. That gives 360 answers per period and 120 per engine. June is the baseline. All numbers are illustrative.

Sheet 1: over time and against rivals

Plannora's AI search performance improved on its baseline in every row. But it still trails Loopcraft by 14.2 points of visibility, and at 360 answers each the margin on that gap is ±6.8. The gap is real.

Sheet 2: engine by engine

The per-engine sheet changes the plan in three ways:

1. Taskwell's Perplexity score is an access choice. Its robots.txt blocks PerplexityBot, so Plannora drops Taskwell from its Perplexity comparison. Beating Taskwell there proves nothing.
2. The Perplexity gap with Loopcraft isn't proven yet. Ten points at 120 answers is inside the margin, so Plannora pools another month before acting on it.
3. ChatGPT and AI Mode are the real gaps. Plannora checks which pages Loopcraft gets cited for in those two engines and plans pages to answer the same prompts.

How to read your AI search performance gaps

The first row matters most: when a whole category rises in the month an engine changes, the rise is drift. The control-prompt test in our GEO measurement guide separates your work from it.

Where Rankbox Fits

Rankbox doesn't track AI citations today, so it won't fill in the benchmark sheet for you. Run the panel by hand with the free AI Visibility Prompt Kit, which writes 30 buyer prompts and a scorecard, or use one of the trackers in our ChatGPT rank tracker roundup.

Where Rankbox helps is lifting AI search performance where the sheet finds gaps. Answer-Space Research maps the questions buyers ask AI in your category, and the Citation-Ready Writer drafts source-backed pages for the prompts your rivals win. The Business plan is $49.50 a month: see pricing.