Robots.txt Generator

Generate robots.txt with the most useful rules to make sure search engines are crawling and indexing your website.

Options

Sitemaps

Must include http(s)://. Separate each sitemap by line. Leave empty if none.

Apply following rules to all crawlers

By default, crawlers should be

Disallow individual paths

Absolute paths to root of website (start with /). Separate each path by line. Leave empty if none.

Code