Robots.txt Generator
Build a valid robots.txt file with rules for any crawler.
Rule 1
Generated robots.txt
User-agent: * Disallow:
What is robots.txt?
A robots.txt file is a plain text file placed at the root of your website (e.g. https://example.com/robots.txt) that tells search engine crawlers which pages and directories they are allowed to visit. It is part of the Robots Exclusion Protocol and is checked by Googlebot, Bingbot, and most other crawlers before they start crawling your site.
Robots.txt does not prevent pages from appearing in search results. If another site links to a page that is blocked in robots.txt, Google may still index that URL based on the link alone. To prevent indexing, use a noindex meta tag instead.
User-agent
Specifies which crawler the rule applies to. Use an asterisk (*) to target all crawlers, or name a specific bot such as Googlebot or Bingbot to apply rules to it alone.
Disallow
Tells the specified crawler not to visit the listed path. Use /admin/ to block a directory, or /page.html to block a specific file. An empty Disallow means everything is allowed.
Allow
Overrides a Disallow rule for a specific path. Useful when you want to block a directory but allow access to one page within it, such as /private/ blocked but /private/sitemap.xml allowed.
Common robots.txt mistakes
A misconfigured robots.txt can accidentally block Googlebot from your entire site. These are the most frequent errors to check before deploying.
Blocking everything with Disallow: /
Adding Disallow: / under User-agent: * tells all crawlers to stay away from every page on your site. This is sometimes left in place after development and causes the entire site to drop out of search results.
Blocking CSS and JavaScript files
Googlebot needs to render your pages correctly to understand them. Blocking /wp-includes/, /assets/, or stylesheet directories can prevent Google from rendering your pages properly and affect how they rank.
Using robots.txt instead of noindex
Robots.txt controls crawling, not indexing. A blocked URL can still appear in search results if it is linked to externally. Use a noindex meta tag or X-Robots-Tag header to reliably prevent indexing.
Missing Sitemap declaration
The Sitemap directive in robots.txt tells crawlers where to find your XML sitemap. Without it, crawlers have to discover your sitemap through other means. Add the full URL of your sitemap to every robots.txt file.
Crawly
Audit your entire site, not just one page
Crawly crawls every URL on your site and surfaces issues automatically. Free to download for Mac.
Download free