Robots.txt Generator and Tester
Build a robots.txt in a minute, decide which AI bots to let in, then test any page against the rules before you upload it.
How to Use the Robots.txt Generator
- Pick a starting point, like WordPress or Online store.
- Add any folders to keep crawlers out of and decide which groups of AI bots to block.
- Switch to the tester tab and try a few important pages with Googlebot and the AI bots.
- Download the file and upload it to the root of your domain.
What Robots.txt Can and Can't Do
robots.txt tells well behaved crawlers which addresses to skip. It doesn't hide pages from search on its own: a blocked page can still show up in Google if other sites link to it, just without a description. To keep a page out of results, let Google crawl it and add noindex instead.
Never block your CSS, JavaScript or image folders. Google needs them to render the page, and blocking them can make your pages look broken to the crawler.
How Matching Works
A crawler follows only the most specific group that names it, and ignores the * group when one does. Within a group, the longest matching rule wins, and Allow wins a tie. That's why an Allow for 1 file can sit inside a blocked folder. The tester follows these rules, so you can check edge cases before they cost you traffic.
Frequently Asked Questions
Where does robots.txt go?
At the root of your domain, so it loads at yourbusiness.com/robots.txt. Each subdomain needs its own.
Should I block AI bots?
Block training bots if you don't want your content used to train models. Keep search bots like OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed if you want to appear and get linked in AI answers.
Does Google follow Crawl-delay?
No. Google ignores Crawl-delay. Some other crawlers still read it.
More Free Tools
See Where Your Site Stands First
Share your website and goals and we will review indexing, pages, speed, your Google Business Profile and competitors by hand, then send a short action plan within two business days.
Want Help Acting on the Results
Tell us about your business and we'll send a free proposal with the fixes that matter most for your leads.