Free Tools
AI Robots.txt Checker
Check if your website allows AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) to access your content. Free, instant results.
Why AI Crawler Access Matters
AI search engines like ChatGPT, Perplexity, Claude, and Google Gemini use web crawlers to index content. If your robots.txt blocks these crawlers, your content won't appear in AI-generated answers, meaning you're invisible to a growing segment of search users.
Which AI Crawlers Should You Allow?
At minimum, allow GPTBot (OpenAI/ChatGPT), ClaudeBot (Anthropic), PerplexityBot, and Google-Extended (Gemini). Blocking these crawlers means your competitors get cited instead of you.
Frequently asked questions
- Can AI bots read my site?
- Enter your domain and this free checker parses your robots.txt and reports which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, and others) are allowed or blocked from reaching your content.
- Which AI crawlers should I allow?
- To be eligible for AI citations, allow at least GPTBot and OAI-SearchBot (ChatGPT), ClaudeBot and Claude-SearchBot (Claude), PerplexityBot (Perplexity), and Google-Extended (Gemini/AI Overviews). The checker lists each one's status.
- Why should I allow AI crawlers in robots.txt?
- Blocking crawlers like GPTBot or PerplexityBot prevents your content from being cited in AI search answers. If you want traffic from ChatGPT, Perplexity, and Gemini, those bots need access.
- Is the robots.txt checker free?
- Yes, free and no signup required.