Free SEO Tool
Robots.txt Generator
Build a valid robots.txt file with rules for any crawler.
Rule 1
Generated robots.txt
User-agent: * Disallow:
Crawly
Audit your entire site, not just one page
Crawly crawls every URL on your site and surfaces issues automatically. Free to download for Mac.
Download free