AI Crawler robots.txt Generator

Every AI vendor runs separate bots for training, search indexing and live page fetches, each with its own robots.txt token. Pick a preset or decide bot by bot, and get a clean robots.txt that keeps you in AI answers while you choose what trains the next model.

  • Instant·Runs in your browser
  • No signup
  • Free

Start from a preset

AI search index

Builds the index an engine cites from. Block it and you drop out of that engine's answers.

Live fetch

Reads a page mid-conversation because a person asked. Each hit is a real question.

Model training

Collects text for future models. Your call: opting out protects content, and the next model knows less about you.

Paths & sitemap

5/5

Search bots allowed

What gets you cited

3/3

Live fetchers allowed

7/7

Training bots allowed

robots.txt
1# robots.txt — generated with rankbox.xyz/tools/ai-robots-txt-generator2 3# Search engines (Google, Bing and everything not named below)4User-agent: *5Allow: /6 7# AI search index8User-agent: OAI-SearchBot9User-agent: Claude-SearchBot10User-agent: PerplexityBot11User-agent: Amazonbot12User-agent: DuckAssistBot13Allow: /14 15# Live fetch16User-agent: ChatGPT-User17User-agent: Claude-User18User-agent: Perplexity-User19Allow: /20 21# Model training22User-agent: GPTBot23User-agent: ClaudeBot24User-agent: Google-Extended25User-agent: Applebot-Extended26User-agent: CCBot27User-agent: Bytespider28User-agent: meta-externalagent29Allow: /30

Upload to the root of your domain so it's reachable at yoursite.com/robots.txt. Bots re-read it on their own schedule, usually within a day.

Next step

Tools that pair with this one

All tools
4 steps

How to use it

  1. 1Choose a preset: Recommended keeps you in AI search and lets you decide on training.
  2. 2Adjust individual bots — they are grouped by job: search index, live fetch, training.
  3. 3Add your sitemap URL and any paths that should stay private.
  4. 4Copy the file and upload it to yoursite.com/robots.txt.
FAQ

Frequently asked

Only the training bots, and only if keeping your content out of future models matters more than being known by them. The search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and the user-triggered fetchers are what get you cited, so blocking those removes you from that engine’s answers.

No. GPTBot collects training data; ChatGPT search uses OAI-SearchBot, and live page reads use ChatGPT-User. You can block GPTBot and stay fully visible in ChatGPT search, which is what the Recommended preset does when you turn training off.

A robots.txt token, not a crawler. It controls whether Google can use your pages to train Gemini and to ground Gemini app answers. It has no effect on Google Search, AI Overviews or AI Mode, which all ride on ordinary Googlebot.

Each crawler re-reads robots.txt on its own schedule, usually within a day. Some user-triggered fetchers (ChatGPT-User, Perplexity-User) state that they may not honor robots.txt at all, because a person asked for the page.
Keep exploring

Let AI crawlers read your site

Key terms

Want this on autopilot?

Enter your site to see how often ChatGPT, Perplexity, Gemini, and Google cite your brand, and exactly what to publish next.

7-day free trial · Card required · Cancel anytime