The Cloudflare Challenge Trap: Are You Silently Blocking ChatGPT from Recommending You?

That matters because of how much of the web sits there. Cloudflare says more than 20% of web domains are behind its network. The visits at stake are small but valuable. Ahrefs reported that over 30 days in mid-2025, AI search sent it 0.5% of its visits but 12.1% of its signups, most of them from ChatGPT. That's one company's data, but it shows why a silent block costs more than it looks.

This guide is the audit. It maps which Cloudflare settings stop which AI bots, shows you where a blocked bot appears in Security Events and AI Crawl Control, and gives you copy-paste custom rules that keep scrapers out while letting verified AI search bots in. For how ChatGPT search finds pages in the first place, see our ChatGPT SEO guide.

What the Cloudflare Challenge Trap Looks Like From ChatGPT's Side

The buyer's prompt

A buyer types "Compare Plannora vs Loopcraft for a 12-person agency" into ChatGPT. Plannora and Loopcraft are made-up brands. ChatGPT decides to search. OpenAI's help page says it rewrites the prompt into targeted queries for its search partners, and it points to Microsoft's privacy statement for them. ChatGPT also surfaces sites that OAI-SearchBot has crawled. And if the buyer asks about a specific page, ChatGPT-User may fetch it live.

Now suppose Plannora's Cloudflare zone challenges both bots. The answer might then read like this. It's an illustrative example, not a captured transcript:

"I could not retrieve information for Plannora, but Loopcraft offers a Team plan with..."

What happens when an AI bot hits a block

OpenAI doesn't publish what ChatGPT says when a page is blocked. Three primary sources do describe what the bots do:

• ChatGPT walks away. Cloudflare tested ChatGPT in August 2025. When it showed ChatGPT a block page, "they again stopped crawling," and no other user agent tried again.
• Answers get built from other sites. In the same post, once Cloudflare also blocked an undeclared crawler it tied to Perplexity, Perplexity fell back on "other data sources," including other websites. The answers "were less specific and lacked details from the original content."
• Anthropic's bots won't fight a challenge. Its crawler page says they "will not attempt to bypass CAPTCHAs."

So a blocked bot doesn't argue. It leaves, and the answer is written from whatever else the engine can read: a review site, an old forum thread, or your competitor's own comparison page.

What the bot actually receives

A Block action returns a 403 with a Cloudflare error page. A rate limit returns a 429. A challenge swaps your page for Cloudflare's, and every challenge response carries the header cf-mitigated: challenge and a text/html body, whatever was requested.

You can see one now, because chatgpt.com itself sits behind Cloudflare. On 28 September 2026, a plain curl -sI https://chatgpt.com/ returned these lines, among others:

HTTP/2 403
cf-mitigated: challenge
server: cloudflare

A browser usually passes that check on its own. A crawler that can't run the challenge gets this response and never sees the page. That's what your site looks like to OAI-SearchBot when a rule challenges it.

Why nobody notices the block

• Your server never sees the request. Block and challenge are terminating actions, so the request stops at Cloudflare. Origin logs look normal.
• The evidence expires. Security Events keeps 24 hours on the Free and Pro plans, and the Free plan shows sampled logs only.
• Nothing drops. Referrals from ChatGPT don't fall, because a blocked page never earned any.
• robots.txt looks fine. A robots.txt check and a firewall are separate layers. Our robots.txt tester can say "allowed" while Cloudflare blocks the same bot.

Why teams set the trap in the first place

Blocking AI bots isn't irrational. Cloudflare estimated for June 2025 that Google crawled sites about 14 times per referral, OpenAI about 1,700 times and Anthropic about 73,000 times. In August 2025 it also accused Perplexity of stealth crawling, which Perplexity denied, saying Cloudflare had misattributed a third-party browser service's traffic.

So blocking training crawlers is a fair business choice. The trap is blocking by broad category, by accident, and catching the search and assistant bots that build buyer answers. When your competitor's page is readable and yours isn't, that's revenue suicide in slow motion.

Which Cloudflare Settings Block Which AI Bots

Cloudflare has many switches that touch AI bots. Some are obvious. Some changed on 15 September 2026 without you touching anything. We call the table below the Cloudflare collateral map: each setting, what it does to each class of bot, and how to carve out an exception.

The columns follow Cloudflare's bot reference, which files OAI-SearchBot, PerplexityBot and Claude-SearchBot as AI Search, and ChatGPT-User, Perplexity-User and Claude-User as AI Assistant. For every bot by name, see AI crawlers.

"Not documented" means Cloudflare's docs don't say. Check Security Events for your own zone before you assume either way.

Bot Fight Mode: the switch with no exceptions

Bot Fight Mode is the Free plan's bot defense. It challenges traffic that matches known bot patterns, which Cloudflare's plan page describes as simple bots from cloud hosting providers and headless browsers. AI bots come from cloud hosting too: a whois lookup on the first range in OpenAI's searchbot.json returns Microsoft, and Perplexity's ranges return Amazon EC2.

The catch is control. Bot Fight Mode runs outside Cloudflare's Ruleset Engine, so Skip rules have no effect on it. Cloudflare says verified bots used to be allowed by default, as Bot Fight Mode reflected. Since 1 July 2026, "Verified" only makes a bot allowable: your Search, Agent and Training choices decide whether it gets in. Anything Cloudflare doesn't verify can't be exempted with a custom rule.

The one documented escape is an IP Access rule, since Bot Fight Mode won't trigger when one matches first. But those rules take IPv4 ranges only as a /24 or /16, or single addresses. OpenAI's searchbot.json listed 39 ranges on 28 September 2026, mostly /28 and /25 blocks. Entered exactly, that's 2 ranges plus 2,048 single addresses, for one bot. If Security Events shows Bot Fight Mode hitting bots you want, the practical fixes are to switch it off or move to Pro.

Super Bot Fight Mode: three groups, one dangerous switch

On Pro and above, Super Bot Fight Mode splits traffic into Definitely automated, Likely automated (Business and up) and Verified bots. Leave Verified bots on Allow. Set it to Block and you stop Googlebot and Bingbot along with every verified AI search bot. Unlike Bot Fight Mode, this one takes exceptions: custom rules run first, and a Skip rule can bypass it.

The 15 September 2026 defaults

This is Cloudflare's second default change in 15 months. On 1 July 2025 it switched its default to blocking AI crawlers, and its press release said every new domain would be asked at sign-up whether to allow them. The setting behind it, Block AI bots, covered training crawlers.

The July 2026 update replaced it with three presets: Search, Agent and Training. Under Security Settings, then Configure AI bot policies, each can be set to Block, Block on pages with ads, or Allow.

For domains added from 15 September 2026, Training and Agent are blocked by default on pages that display ads. Search stays allowed. Cloudflare's own example of an Agent is ChatGPT-User. So on a new zone, OAI-SearchBot can still crawl your ad-carrying pages, but ChatGPT-User can't fetch them live when a buyer asks about them.

How does Cloudflare know a page shows ads? Its July 2025 write-up says it scans HTML for ad-unit code, using about 400 patterns from ad-blocker filter lists, and matches requests against hostnames it knows serve ads. If a page runs ad-network code, assume the Agent default can apply to it.

Training blocks now reach Googlebot and Bingbot

Cloudflare's Block AI Bots page says that from 15 September 2026, crawlers that combine Search and Training "will also be blocked by all configurations to block AI training, including the legacy 'Block AI bots' option." The July blog names Googlebot, Applebot and BingBot as examples. Customers could opt out up to 15 September.

If you switched on Block AI bots in 2025 to keep GPTBot out and didn't opt out, check Bingbot today. Bing's index grounds Copilot, and Microsoft is one of ChatGPT's search partners. The safer way to block training is by name: in AI Crawl Control, set GPTBot, ClaudeBot, CCBot and other trainers to Block one by one. Blocking GPTBot doesn't touch ChatGPT search.

Challenges, Turnstile and Under Attack mode

People often say a bot "hit Turnstile." Strictly, Turnstile is a widget that runs on a page after the visitor has already arrived. It gates actions like submitting a form. It doesn't stop a crawler from reading your pricing page.

What stops crawlers is a full-page challenge, served by a WAF rule, a bot setting, a rate limit or Under Attack mode. Under Attack mode shows a Managed Challenge to every visitor, and the page needs JavaScript to pass. Browser Integrity Check, on by default, challenges visitors with a missing or non-standard user agent. Cloudflare doesn't document how either treats verified AI bots.

The 20-Minute Cloudflare Audit

Run these steps in one sitting. Security Events only keeps 24 hours on lower plans, so steps 4 and 5 need to happen on the same day.

1. Write down your bot product. Go to Security, then Settings, and filter by Bot traffic. Free plans have Bot Fight Mode. Pro and up have Super Bot Fight Mode. Note every switch that's on, then clear the filter and note Browser Integrity Check too.
2. Read your AI bot policies. On the same page, open Configure AI bot policies and note what Search, Agent and Training are set to. If Training is blocked, or the legacy Block AI bots toggle is on, add Googlebot and Bingbot to your checks in step 4.
3. Open AI Crawl Control. On the Crawlers tab, find OAI-SearchBot, ChatGPT-User, PerplexityBot and Perplexity-User. Check the Action column, and compare each bot's requests with its unsuccessful ones. On the Metrics tab, filter to one crawler and read the status code chart. A wall of 4xx means blocks or challenges, though 404s count too.
4. Open Security Events. Go to Security, then Analytics, and select the Events tab. In Sampled logs, choose Edit columns and add User agent and Path. Scan Top events by source for user agents that contain a bot name, then expand an event. Its Service field names the feature that acted, such as Bot Fight Mode, Super Bot Fight Mode, Custom rules or Rate limiting rules.
5. Run the ask-and-watch test. In ChatGPT, ask it to open one specific URL on your site and summarize it. Do the same in Perplexity. OpenAI says ChatGPT "may" fetch a page with ChatGPT-User when you ask. Within a few minutes, look for the fetch in Security Events and AI Crawl Control.
6. Check the response headers. Run the command below with a bot user agent. A cf-mitigated: challenge line, or a 403 from server: cloudflare, means Cloudflare stopped the request. Run it again without -A to see if the user agent is the trigger.
7. Read your origin logs. If they show zero OAI-SearchBot hits for weeks, something upstream is eating them. Our free AI crawler log analyzer counts each bot's hits and errors in your browser.

Here's the header check from step 6, using OpenAI's published ChatGPT-User string:

curl -s -o /dev/null -D - \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot" \
  https://yoursite.com/pricing | grep -i -E "^(HTTP|server:|cf-mitigated)"

Read the result with care. Your laptop's IP isn't OpenAI's, so Cloudflare won't treat you as a verified bot. A clean result only proves that no rule blocks that user agent from your address. Once lane 2 of the rulebook below is live, this test should return a 403, because to Cloudflare you're an impostor. The real verdict comes from steps 3 to 5.

What a blocked OAI-SearchBot looks like

In Sampled logs, a challenged OAI-SearchBot request looks like this once you expand it. The rule name is illustrative, since custom rule names are yours, and large volumes are sampled, so not every request appears:

Action:      Managed Challenge
Service:     Custom rules  (rule: "Challenge cloud ASNs")
User agent:  Mozilla/5.0 (...) compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
Path:        /compare/loopcraft

A blocked ChatGPT-User looks the same, with ChatGPT-User/1.0; +https://openai.com/bot at the end of the user agent. In AI Crawl Control, both show up as unsuccessful requests. If the Action column there says Allow but requests still fail, Cloudflare's AI Crawl Control guide points to rules that run before its own.

The Three-Lane Rulebook for Cloudflare Custom Rules

The Three-Lane Rulebook is our method for blocking scrapers without blocking the bots that recommend you. Its logic fits in one line: let in by proof, block by claim, then defend against everything else.

• Lane 1: let in by proof. Skip your bot defenses for requests that Cloudflare has verified as AI search, AI assistant or search engine bots, or that pair a known user agent with the vendor's published IPs.
• Lane 2: block by claim. Block anything that claims to be one of those bots in its user agent but can't prove it.
• Lane 3: everything else. Your normal scraper defenses stay on for all remaining traffic.

Order matters. Cloudflare runs custom rules in order, and custom rules run before rate limiting and Super Bot Fight Mode. Put lane 1 at the top.

Lane 1: Skip verified AI search and assistant bots

Create a custom rule, choose Edit expression, and paste:

(cf.verified_bot_category in {"AI Search" "AI Assistant" "Search Engine Crawler"})
or (ip.src in $ai_search_bots and (
  http.user_agent contains "OAI-SearchBot"
  or http.user_agent contains "ChatGPT-User"
  or http.user_agent contains "PerplexityBot"
  or http.user_agent contains "Perplexity-User"))

Set the action to Skip, and tick All remaining custom rules, All rate limiting rules, All Super Bot Fight Mode rules (Pro and up) and Browser Integrity Check. Leave managed rules running, and keep Log matching requests on so you can see the rule work.

Why two halves? cf.verified_bot_category only matches bots Cloudflare has verified, using the documented category strings. But Cloudflare keeps "AI Search" only for backward compatibility, and verified status can change. It removed Perplexity from its list in August 2025, and its docs don't say whether that status came back. The second half uses each vendor's published IP ranges, which don't depend on Cloudflare's verdict.

Perplexity's own docs recommend the same user agent plus IP pairing, but tell you to set the action to "Allow." Current custom rules have no Allow action. Skip is the equivalent.

ChatGPT's cloud browser, formerly ChatGPT agent, signs its requests with Web Bot Auth. Cloudflare calls such bots signed agents, and since 1 July 2026 it counts them as verified. On Enterprise with Bot Management, you can also name it directly: OpenAI lists its Cloudflare detection ID as 129220581. Add this line to lane 1:

or any(cf.bot_management.detection_ids[*] eq 129220581)

Lane 2: Block impostors

Any client can send a famous bot's user agent, so rules that trust the name alone are easy to slip past. This rule blocks anyone who claims to be one of these four bots but isn't verified and isn't in the list:

(http.user_agent contains "OAI-SearchBot"
  or http.user_agent contains "ChatGPT-User"
  or http.user_agent contains "PerplexityBot"
  or http.user_agent contains "Perplexity-User")
and not cf.client.bot
and not ip.src in $ai_search_bots

Set the action to Block. One rule of thumb: only add a user agent to lane 2 after that vendor's IPs are in your list. Otherwise you'll block the real bot the day Cloudflare's verdict on it changes.

Lane 3: Keep your scraper defenses

Lanes 1 and 2 only handle traffic that names a known bot. Everything else meets your normal defenses:

• Pro and Business: set Super Bot Fight Mode's Definitely automated to Managed Challenge or Block, Likely automated (Business) to Managed Challenge, and Verified bots to Allow.
• Enterprise: use Cloudflare's bad-bots pattern, (cf.bot_management.score lt 30 and not cf.bot_management.verified_bot), with a Managed Challenge.
• Training crawlers: block them by name in AI Crawl Control, not with the Training preset, to leave Googlebot and Bingbot alone.
• Rate limits: add a condition that Verified Bot is false. Cloudflare warns that limiting verified bots can affect SEO.
• Old custom rules: add and not cf.client.bot to any rule that challenges by country or ASN, as Cloudflare's own example does.

Build and refresh the IP list

Create one custom IP list named ai_search_bots. On 28 September 2026 the four vendor files held 39 ranges for OAI-SearchBot, 230 for ChatGPT-User, 8 for PerplexityBot and 4 for Perplexity-User. That's 281 items, well under the Free plan's 10,000.

The files change. The creationTime in chatgpt-user.json read 25 September 2026, so a list you typed by hand in August may already be stale. This script rebuilds the list from the source files through Cloudflare's Lists API, using an API token that can edit lists. Run it daily from any scheduler:

#!/usr/bin/env bash
# Rebuild the ai_search_bots list from the vendors' published IP files.
set -euo pipefail
for url in https://openai.com/searchbot.json \
           https://openai.com/chatgpt-user.json \
           https://www.perplexity.com/perplexitybot.json \
           https://www.perplexity.com/perplexity-user.json; do
  curl -sfL "$url"
done | jq -s '[.[].prefixes[] | {ip: (.ipv4Prefix // .ipv6Prefix)}]' > items.json
 
# An empty PUT deletes every item, so stop if nothing came back.
test "$(jq length items.json)" -gt 0
 
curl -sf -X PUT \
  "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/rules/lists/$LIST_ID/items" \
  -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
  -H "Content-Type: application/json" \
  --data @items.json

If any file fails to download, the script stops before the upload, so a half-built list never replaces a good one. The PUT swaps in the whole list, dropping ranges a vendor has retired.

The rulebook by Cloudflare plan

Mistakes that reopen the trap

• Adding "AI Crawler" to lane 1. AI Crawl Control enforces its blocks with a custom rule at the end of your list. A Skip rule above it for the AI Crawler category would let GPTBot straight past your own block. The same goes for any AI Assistant bot you block there: drop that category from lane 1 and name the bots you want instead.
• Trusting a Skip rule to beat the presets. Cloudflare doesn't document whether a Skip rule overrides the Search, Agent and Training presets. Set the preset itself.
• Letting the list go stale. New ranges from OpenAI or Perplexity fall into lane 2 until your list catches up. Automate the refresh.

Worked Example: Plannora's Hidden Block

Plannora is a made-up project management tool on Cloudflare's Pro plan. The numbers are illustrative, but the arithmetic is real. The team runs the audit after a sales rep notices ChatGPT pointing prospects to Loopcraft's comparison page.

They find two settings, each reasonable on its own:

1. A 2025 toggle. Someone switched on Block AI bots to keep GPTBot out. Nobody opted out before 15 September, so it now blocks mixed-purpose crawlers too. Security Events shows Bingbot blocked.
2. An incident rule. During a scraping spike, an engineer added "Challenge cloud ASNs," a custom rule with a Managed Challenge and no bot exception.

AI Crawl Control shows seven days before the fix and seven days after:

Before the fix, 0 of 2,160 requests reached a page. After it, 2,354 of 2,391 did, about 98%. The 37 failures left are 404s from retired URLs, which the team redirects the same day.

The fix took four steps, in this order:

1. Add lane 1 at the top of the custom rules, then lane 2 below it.
2. Add and not cf.client.bot to "Challenge cloud ASNs," so it stops catching verified bots even if lane 1 is ever edited.
3. Set the Training preset to Allow, along with the legacy Block AI bots toggle where the zone still shows it, and block GPTBot, ClaudeBot and CCBot by name in AI Crawl Control.
4. Rerun the ask-and-watch test and confirm the live fetches in Security Events.

Unblocking doesn't win the recommendation on its own. It puts Plannora's pages back in the pool the answer is written from. What gets named after that is a content question: see how to rank on ChatGPT.

Beyond Cloudflare: Vercel, AWS WAF, Akamai and Fastly

The same trap exists on other platforms, with different switches:

• Vercel. Its Bot Protection ruleset challenges non-browser traffic but skips verified bots. The AI Bots ruleset covers crawling "for training data, search purposes, or user-generated fetches," so Deny can block search bots like OAI-SearchBot too. Both are off by default.
• AWS WAF. In the Bot Control rule group, the CategoryAI rule blocks by default and "applies the action to all matches, regardless of whether the bots are verified or unverified." Override that rule if you want AI search bots in.
• Akamai. On 3 September 2026, Akamai split its AI Bots category into AI training crawlers, AI search crawlers and AI fetchers and agents. If your policy predates the split, check the action on each new category.
• Fastly. Its bot variables separate AI crawlers from AI fetchers, but it defines AI crawlers as bots for "building AI models or indexes." A rule on that flag alone can catch search indexers, so pair it with the verified-bot flag.

On any platform, the audit is the same: find the bot's requests in the firewall log, read which rule acted, then allow by proof, not by name. Our two-engine site checklist covers the rest of the access work.

Where Rankbox Fits Once the Door Is Open

Rankbox works on the other side of the firewall. It doesn't manage Cloudflare, your WAF or robots.txt, and it doesn't track AI citations today. For access checks, use the free tools linked above, or run a page through the AI search readiness check.

What Rankbox does is fill the pool you just reopened. Answer-Space Research maps the questions buyers ask ChatGPT, Perplexity and Google, like "Plannora vs Loopcraft for agencies." The Citation-Ready Writer researches the live web and writes source-backed articles of 2,000 to 3,500 words, which reach your site through Rankbox's API. The Business plan is $49.50 a month with a 7-day trial: see pricing.