robots.txt Tester for AI & Search Crawlers
Paste your robots.txt, enter a URL and pick a user agent. The tester applies the same longest-match rules Google and the AI vendors use and tells you exactly which line allowed or blocked the request, so you catch a bad rule before a crawler does.
- Instant·Runs in your browser
- No signup
- Free
robots.txt
3 groups · 7 rules · 1 sitemap
Test
Blocked
GPTBot → /blog/how-to-rank
- Group
- gptbot, claudebot (line 7)
- Rule
- Line 9: Disallow: / is the most specific match.
Every known bot
For this path: 15 allowed, 2 blocked
- GooglebotGoogleline 2
- BingbotMicrosoftline 2
- OAI-SearchBotOpenAIline 2
- ChatGPT-UserOpenAIline 2
- GPTBotOpenAIline 9
- Claude-SearchBotAnthropicline 2
- Claude-UserAnthropicline 2
- ClaudeBotAnthropicline 9
- PerplexityBotPerplexityline 12
- Perplexity-UserPerplexityline 2
- Google-ExtendedGoogleline 2
- Applebot-ExtendedAppleline 2
- CCBotCommon Crawlline 2
- BytespiderByteDanceline 2
- meta-externalagentMetaline 2
- AmazonbotAmazonline 2
- DuckAssistBotDuckDuckGoline 2
Next step
Tools that pair with this one
- InstantAI Crawler Log AnalyzerSee which AI bots visit your site, and which pages they want.AI crawlersOpen
- Live checkAI Search Readiness CheckScan any site for the 12 things AI engines need to cite it.AI crawlersOpen
- Instantllms.txt GeneratorWrite the llms.txt file AI assistants read first.AI crawlersOpen
4 steps
How to use it
- 1Paste the contents of your robots.txt (or use the sample).
- 2Enter the full URL or path you want to test.
- 3Pick a user agent, or type a custom one.
- 4Read the verdict and the exact rule that decided it; fix and re-test.
FAQ
Frequently asked
The crawler picks the group whose User-agent line best matches its name (a specific name beats *), then within that group applies the most specific rule — the longest matching path. When an Allow and a Disallow are equally long, Allow wins. That is the algorithm in RFC 9309 and Google’s documentation, and the one this tester uses.
Yes. A bot uses its own group if one exists, otherwise the * group. It never combines the two, which is a common surprise: rules under * do not apply to GPTBot if GPTBot has its own group.
Google, Bing and the major AI crawlers support * (any characters) and $ (end of URL) in paths. This tester supports both. Some smaller crawlers only support prefix matching.
Keep exploring
Get recommended by Siri and voice assistants
- FeatureCitation-Ready WriterDeeply researched, source-backed articles AI loves to quote.
- FeatureCitation TrackingSee where AI answers quote and recommend your brand.
- AI toolRankbox for ChatGPTPlan content in ChatGPT with Rankbox's research tools.
- AI SEO guideGemini SEO guideGet grounded into Gemini's answers
- PlaybookApple Intelligence & Siri: How iOS 18/26 Routes Queries to ChatGPTHow Apple Intelligence and Siri route questions, and how to be the recommendation.
- PlaybookWhat Does Apple Intelligence Do? Every Feature Explained for 2026Every Apple Intelligence feature in 2026, with devices, languages and privacy.
Want this on autopilot?
Enter your site to see how often ChatGPT, Perplexity, Gemini, and Google cite your brand, and exactly what to publish next.
7-day free trial · Card required · Cancel anytime