AI Crawler Checker
Find out if ChatGPT, Claude, Perplexity and Google's AI features can read your pages, and which line in your robots.txt decides it.
How to Use the AI Crawler Checker
- Enter your homepage or any page you care about, like a service page.
- We fetch your robots.txt, llms.txt and the page itself.
- Each AI crawler is matched against your rules the way Google documents robots.txt matching, so you see allowed or blocked and why.
What Blocking Costs You in AI Answers
When someone asks ChatGPT or Perplexity to recommend a business, those tools can only cite pages their crawlers were allowed to read. A single Disallow line, often added by a security plugin or a template, can quietly keep you out of every answer.
The bots fall into 3 groups. Search and answer bots read pages so they can show and link them. User fetchers visit when a person pastes your link into a chat. Training bots collect pages to train future models. You can block training and still appear in AI answers, which is the setup most businesses want.
What This Check Can't See
robots.txt is a request, not a lock. Your host or CDN can still turn bots away at the network level, for example with an AI bot blocking switch in Cloudflare. If a crawler shows as allowed here but your server logs never show it, check those settings next.
Googlebot handles both regular Google results and AI Overviews, so there's no way to block one and keep the other. Google-Extended only controls whether Google uses your content for Gemini, and Google says it doesn't affect Search.
Frequently Asked Questions
Should I block GPTBot?
Blocking GPTBot stops OpenAI from using your pages to train models. It doesn't stop ChatGPT search from finding you, which uses OAI-SearchBot. If you want to show up in ChatGPT answers, keep OAI-SearchBot allowed.
Does llms.txt help me rank in AI answers?
It might not. llms.txt is a proposal for a plain text summary of your site, and support from AI companies is unclear. It takes 10 minutes to make, so it's cheap to add, but don't expect it to replace good pages and clear robots.txt rules.
Why does a bot show as blocked when I never blocked it?
Usually a rule under User-agent: * blocks a folder the page lives in, or a plugin added a group for that bot. The table shows the exact line number so you can find it.
More Free Tools
See Where Your Site Stands First
Share your website and goals and we will review indexing, pages, speed, your Google Business Profile and competitors by hand, then send a short action plan within two business days.
Want Help Acting on the Results
Tell us about your business and we'll send a free proposal with the fixes that matter most for your leads.