Cloudflare AI Bot Management: Blocking Scrapers vs. Preserving Citations

The old one-click switch is on its way out. Cloudflare launched it in July 2024 as a toggle labeled "AI Scrapers and Crawlers", later renamed it Block AI bots, and on 15 September 2026 announced its deprecation in favor of the three controls. Zones that had it switched on were moved to new settings, and one of those moves can cost you live citations from ChatGPT and Perplexity on pages that carry ads.

Most sites only want to stop one thing. Cloudflare says fewer than 1% of its sites block Search bots, while 17% use some way to block training. This guide to Cloudflare AI bot management walks the settings screen by screen, maps each one to the bots it reaches, and gives you a decision path by goal. For firewall rules and challenge problems, see the Cloudflare challenge trap.

Two shorter guides go deeper on single questions: how Cloudflare classifies AI bots and where to see them, and the step-by-step way to block AI bots in Cloudflare without losing AI search traffic.

What Changed in Cloudflare AI Bot Management on 15 September 2026

Cloudflare rebuilt its AI controls twice in 2026. On 1 July it replaced the single training block with three presets, so any customer could treat search crawlers, live assistants and training crawlers differently. On 15 September it changed what "Block" means for training and added a softer option.

The plan's toggle, and what it really did

The worry behind this post is a one-click switch that quietly removes you from AI answers. Here's the record as of 1 October 2026:

• July 2024: a toggle called AI Scrapers and Crawlers, under Security, then Bots, on every plan.
• July 2025: a new choice between blocking everywhere or only on pages with ads. Cloudflare's docs call the setting Block AI bots.
• Up to September 2026: Cloudflare's Block AI Bots page says the legacy option blocked verified bots classified as crawling for AI training, plus similar unverified bots, and excluded bots that do both search and training.

So the toggle, as documented, aimed at training crawlers. It wasn't documented to block OAI-SearchBot or PerplexityBot, which Cloudflare files as search. The real risk to citations now sits in the controls that replaced it, and in the migration.

How existing zones were migrated

Cloudflare's 15 September post gives two migration tables. For zones that only ever used the old toggle:

Zones that had already set the three controls keep their Search and Agent choices. A Training choice of Block, or Block on pages with ads, became Disallow AI Training. The post says most owners need to do nothing. On the training side of Cloudflare AI bot management, that's true. It isn't true for the Agent row if your pages carry ads and you want ChatGPT to read them on request.

What new domains get

From 15 September, Cloudflare offers new domains one of two presets during onboarding, depending on whether the site earns money from ads:

You can change any of these during onboarding or later. One caution on sources: on 1 October 2026, Cloudflare's Block AI Bots docs page still read "Last updated Jul 1, 2026" and described the July plan for Cloudflare AI bot management defaults. A headless Chrome render showed the same text. The 15 September blog post is newer, so this guide follows it.

The Settings Screen, Control by Control

Cloudflare AI bot management lives in two places in the dashboard: Security Settings, filtered to Bot traffic, and the AI Crawl Control section. Here's what each screen holds and which bots it reaches.

Search, Agent and Training

Cloudflare defines the three behaviors on its bots concept page:

• Search collects or indexes your content so it can answer questions about it later. Cloudflare no longer separates AI search from classic search, so this covers OAI-SearchBot and Claude-SearchBot as well as search engines.
• Agent acts in real time for a person, such as chat fetch bots and browser-use agents. Cloudflare's July post names ChatGPT-User as an example.
• Training crawls content to train or fine-tune a model, including crawlers that also do search.

Cloudflare's docs say each blocking option stops verified bots classified with that behavior, plus similar unverified bots. Per the 15 September post, each control takes Allow, Block on pages with ads, or Block. Training alone adds Disallow AI Training. That option publishes a no-training line in robots.txt, keeps "Accountable" mixed-use crawlers in for search, and blocks every other training crawler, including the training-only crawlers run by Amazon, Anthropic, Meta and OpenAI. Cloudflare counts Apple, Google and Microsoft as Accountable.

There's no Disallow on pages with ads. Cloudflare says the list of ad pages is too large and changes too often to write into robots.txt. There's also no Disallow for Agent yet.

Bot Preference Sync, managed robots.txt and Content Signals

Bot Preference Sync, announced on 21 August 2026 for all plans, writes your Search, Agent and Training choices into robots.txt. It prepends its block to any file you already serve, so your own rules stay. It replaces the older managed robots.txt, which Cloudflare's docs still list under the name "Set your preference to block training in robots.txt."

Both add Content Signals, a comment block plus a line such as Content-signal: search=yes, ai-train=no, use=reference. These are statements of preference. A crawler that ignores robots.txt ignores them too, which is why the edge block matters. Zones on the Free plan that serve no robots.txt of their own show Cloudflare's Content Signals Policy text by default.

AI Crawl Control

AI Crawl Control is on every plan. Its Crawlers tab lists each AI crawler with its category, requests and robots.txt violations, and an Action you can set to Allow or Block. Cloudflare's management guide says:

• On Free, it spots crawlers by user agent only. Paid plans can use Bot Management detection IDs.
• Each block adds to one WAF custom rule named "AI Crawl Control," placed after your other custom rules.
• Paid plans can choose a 403 or a 402 response for blocked crawlers.

The Directives tab, renamed from Robots.txt in April 2026, lists crawlers that requested paths your robots.txt disallows. That's the fastest way to find a bot that ignores your file.

Pay per crawl and Pay Per Use

Pay per crawl answers crawlers with a 402 and a price, and lets a paying crawler through with a 200. It was still in closed beta on 1 October 2026. Any WAF or Bot Management block overrides its "charge" action, so a blocked crawler never reaches the payment step. On 30 September, Cloudflare put Pay Per Use into beta, where AI companies pay when they use content rather than when they crawl it.

Verified bots and signed agents

Cloudflare's verified bots page sets two bars: honest identity (a Web Bot Auth signature, a published IP list with a stable user agent, or reverse DNS) and no abuse, including obeying robots.txt. Since 1 July 2026, signed agents count as verified too. The big change is what "verified" means. Cloudflare's July post says verified no longer means allowed by default. It makes a bot allowable, and your Search, Agent and Training choices decide.

Bot Fight Mode and Super Bot Fight Mode

Bot Fight Mode is the Free plan's switch. It challenges traffic that matches known bot patterns, and Cloudflare's docs say Skip rules have no effect on it. Super Bot Fight Mode on Pro and up adds groups for definitely automated, likely automated (Business and up) and verified bots, and it does accept Skip rules. Neither is part of Cloudflare AI bot management as such, but both meet AI crawlers that Cloudflare doesn't verify.

Cloudflare AI Bot Management Settings at a Glance

This table is the whole Cloudflare AI bot management screen on one page, as of 1 October 2026. "Hits" uses Cloudflare's behavior labels.

Two rows need a note. AI Labyrinth only traps crawlers that ignore no-crawl instructions, and its events are not mitigations. Redirects for AI Training sends verified training crawlers to your canonical URL, while search engines and AI assistants see the page unchanged.

Which Bots Each Cloudflare AI Bot Management Control Reaches

Cloudflare AI bot management acts on behaviors, not names. This table maps the bots that decide AI answers to the control that can stop them. Categories come from Cloudflare's bot reference and its Radar directory. For every user agent and IP list, see our AI crawler directory.

The Perplexity gap

On 1 October 2026, the AI bot transparency table on Cloudflare Radar's AI Insights listed Perplexity as not verified and ByteDance as failing all three checks. OpenAI, Anthropic, Meta and Amazon passed all three. That matters for Perplexity in two ways. A "Verified bots: Allow" setting doesn't cover PerplexityBot. And any defense that challenges unverified automation can meet it. Perplexity's own crawler docs ask you to allow its published IP ranges in your WAF. Our challenge trap guide has a rule that pairs its user agent with those IPs.

Live fetchers carry the most risk

Radar's directory page for ChatGPT-User showed 31.3% of its requests across Cloudflare getting a 403 in the seven days to 1 October 2026. For OAI-SearchBot the figure was 12.8%. Those are network-wide numbers, not yours. They show where blocks pile up: on the bot that reads your page while a buyer waits.

The Citation-Safe Path: Settings by Goal

The Citation-Safe Path is our decision table for Cloudflare AI bot management. Pick the row that matches what you want, set the controls it lists, and read the last column before you save. That column is what you give up.

How to pick a row

1. No ads on the pages you want quoted? Row 1 or row 2 fits most SaaS, ecommerce and local service sites.
2. Training worries you more than discovery? Row 2 stops training crawlers without touching a search engine.
3. Ads pay the bills? Row 3 protects ad views. Keep ad code off the pages you want assistants to read.
4. Content worth paying for? Row 4 is a beta. A search bot that won't pay drops out of answers that would have cited you.
5. Thinking about row 5? Fewer than 1% of Cloudflare sites block Search. If you only meant to stop scrapers, go back to row 2.

Three settings that look safe and aren't

• Training on Block. It reads like a stronger Disallow. Since 15 September it also blocks Googlebot, Bingbot and Applebot from your whole site.
• Search on Block to stop "AI search scrapers." Cloudflare treats AI search and classic search as one behavior, so the block reaches search engines too.
• NOARCHIVE to opt out of Bing training. Until Bing reads Cloudflare's robots.txt line, Cloudflare points to NOARCHIVE. But Bing's webmaster guidelines say NOARCHIVE "prevents content from being used in Copilot responses and grounding results." Use it only on pages you don't need quoted in Copilot.

Row 2 has one more trade-off. Google says Google-Extended controls training and "grounding" in Gemini Apps, while having no effect on Google Search. If your synced robots.txt disallows Google-Extended, expect less use of your pages in Gemini app answers. Search, AI Overviews and AI Mode aren't affected. For hand-written templates for each goal, see our AI crawler robots.txt guide.

Worked Example: Tallyfold's Two-Setting Mistake

Tallyfold is a made-up invoicing app for agencies, on Cloudflare's Pro plan at tallyfold.example. Its free invoice-template gallery carries display ads that fund the free tier. The numbers below are illustrative, but the arithmetic is real.

How the zone drifted

Tallyfold's Cloudflare AI bot management changed three times in a year, and only one change was deliberate.

1. 2025: marketing switched on Block AI bots, set to block everywhere, to keep GPTBot out.
2. 15 September 2026: Cloudflare's migration set Training to Disallow AI Training, which was fine, and Agent to Block on pages with ads, which nobody noticed.
3. 22 September 2026: an engineer read "Disallow" as weaker than the old block and changed Training to Block.

What AI Crawl Control and Security Analytics showed

The team pulled seven days of data, 23 to 29 September:

Every failed live fetch hit a template page with ads. That's 2,080 of 3,200 assistant reads lost, or 65%. Googlebot failed on every request after the 22 September change. OAI-SearchBot was fine, because Search stayed on Allow. Its 19 failures were 404s.

The fix, in order

1. Set Training back to Disallow AI Training. Googlebot, Bingbot and Applebot crawl again, and GPTBot and ClaudeBot stay blocked.
2. Set Agent to Allow. The template gallery is Tallyfold's biggest signup source, so an assistant reading it is worth more than an ad view.
3. Open tallyfold.example/robots.txt and confirm the synced block lists the training tokens it expects.
4. Leave NOARCHIVE off the templates, so they stay eligible for Copilot answers.

The next week, the three assistant bots made 3,370 requests and 50 failed, all 404s from retired templates. That's a 98.5% success rate, up from 35.0%. Googlebot was back to normal within a day.

Fixing Cloudflare AI bot management doesn't win citations on its own. It puts Tallyfold's pages back in the pool that answers are written from. Our guide to measuring AI referral traffic in GA4 covers where those visits show up.

Where Rankbox Fits Once the Settings Are Right

Rankbox doesn't run a CDN, manage Cloudflare settings or robots.txt for you, and it doesn't track AI citations today. For access checks, use our free AI search readiness check and AI crawler log analyzer, or draft a file with the AI crawler robots.txt generator, which groups bots by job: search, live fetch and training.

What Rankbox does is write the pages your newly allowed bots will read. Answer-Space Research maps the questions buyers ask ChatGPT, Perplexity and Google, and the Citation-Ready Writer turns them into source-backed articles of 2,000 to 3,500 words that reach your site through Rankbox's API. The Business plan is $49.50 a month with a 7-day trial: see pricing.