robots.txt Tester for AI & Search Crawlers
Paste your robots.txt, enter a URL and pick a user agent. The tester applies the same longest-match rules Google and the AI vendors use and tells you exactly which line allowed or blocked the request, so you catch a bad rule before a crawler does.
- Instant·Runs in your browser
- No signup
- Free
robots.txt
3 groups · 7 rules · 1 sitemap
Test
Blocked
GPTBot → /blog/how-to-rank
- Group
- gptbot, claudebot (line 7)
- Rule
- Line 9: Disallow: / is the most specific match.
Every known bot
For this path: 15 allowed, 2 blocked
- GooglebotGoogleline 2
- BingbotMicrosoftline 2
- OAI-SearchBotOpenAIline 2
- ChatGPT-UserOpenAIline 2
- GPTBotOpenAIline 9
- Claude-SearchBotAnthropicline 2
- Claude-UserAnthropicline 2
- ClaudeBotAnthropicline 9
- PerplexityBotPerplexityline 12
- Perplexity-UserPerplexityline 2
- Google-ExtendedGoogleline 2
- Applebot-ExtendedAppleline 2
- CCBotCommon Crawlline 2
- BytespiderByteDanceline 2
- meta-externalagentMetaline 2
- AmazonbotAmazonline 2
- DuckAssistBotDuckDuckGoline 2
Next step
Tools that pair with this one
- Instantllms.txt GeneratorWrite the llms.txt file AI assistants read first.AI crawlersOpen
- InstantAI Crawler robots.txt GeneratorAllow search bots, decide on training bots, one file.AI crawlersOpen
- InstantAI Crawler Log AnalyzerSee which AI bots visit your site, and which pages they want.AI crawlersOpen
How to use it
- 1Paste the contents of your robots.txt (or use the sample).
- 2Enter the full URL or path you want to test.
- 3Pick a user agent, or type a custom one.
- 4Read the verdict and the exact rule that decided it; fix and re-test.
Frequently asked
How does robots.txt decide between Allow and Disallow?
The crawler picks the group whose User-agent line best matches its name (a specific name beats *), then within that group applies the most specific rule — the longest matching path. When an Allow and a Disallow are equally long, Allow wins. That is the algorithm in RFC 9309 and Google’s documentation, and the one this tester uses.
Does a bot that isn't named fall back to the * group?
Yes. A bot uses its own group if one exists, otherwise the * group. It never combines the two, which is a common surprise: rules under * do not apply to GPTBot if GPTBot has its own group.
Do wildcards work in robots.txt?
Google, Bing and the major AI crawlers support * (any characters) and $ (end of URL) in paths. This tester supports both. Some smaller crawlers only support prefix matching.
Want this on autopilot?
Enter your site to see how often ChatGPT, Perplexity, Gemini, and Google cite your brand, and exactly what to publish next.
No credit card required · Free 7-day trial