Robots.txt Generator

Control how search engine bots crawl your website. Hide sensitive folders, declare your Sitemap location, and protect your SEO ranking.

General Settings

Default Access (All Bots)

Crawl-Delay (Seconds)

How many seconds bots should wait between requests (Reduces server load).

Sitemap URL

Restricted Directories

Quick Presets

Your Robots.txt File

User-agent: *

What is Robots.txt?

Robots.txt is a simple text file located in your website's root directory. It tells search engine crawlers (spiders) which parts of your site they should or should not scan. It is a cornerstone of technical SEO.

Meaning of User-agent: *

This command targets "All Bots". Any rule written below it, such as `Disallow: /admin/`, applies to every visitor bot from Google to Yandex.

Why add a Sitemap?

Adding `Sitemap: https://...` at the bottom of your robots.txt allows Google to discover new content on your site much faster. Even if you don't use Search Console, bots can find your map here.

Robots.txt FAQ

Where should robots.txt be placed?

It must be in the root directory. It should be accessible at `site.com/robots.txt`. If placed in a subdirectory (site.com/blog/robots.txt), it will not work.

Does Disallow remove pages from Google?

No, it only prevents crawling. If the page is already indexed or linked externally, it may still appear in results. Use the "noindex" meta tag for removal.