Free Robots.txt Generator
The Robots.txt Generator builds a robots.txt file that tells search-engine crawlers which parts of your site to crawl or avoid. Add your rules and sitemap, then copy the result.
How to Use
Enter the paths to disallow or allow (one per line), set the user-agent, add your sitemap URL, and the robots.txt generates live.
- One rule per line for allow/disallow
- Optional sitemap reference
- Copy to your site root
Related SEO & Webmaster Tools
You might also need
Tools from other categories that people pair with the Robots.txt: