Robots.txt Generator

Generate a valid robots.txt

The Robots.txt Generator builds a correctly formatted robots.txt file that tells search engine crawlers what they can and can't access on your site. Choose to allow everything, block everything, or specify exact paths to disallow, add a crawl-delay, and link your sitemap — then copy the result into a robots.txt file at your site's root.

What is robots.txt?

The robots.txt file lives at the root of your domain (e.g. example.com/robots.txt) and is the first thing well-behaved crawlers check. It uses simple directives — User-agent to target a crawler, Disallow to block paths, Allow to permit them, and Sitemap to point to your sitemap. It guides crawling, helping search engines focus on the pages that matter.

How to use it

1. Pick a mode: Allow all, Block all, or Custom rules.
2. For custom, list the paths to disallow (one per line).
3. Add your sitemap URL and optional crawl-delay, then copy the file.

Common directives

Directive	Meaning
User-agent: *	Applies to all crawlers
Disallow: /admin	Block a path from crawling
Allow: /	Permit crawling
Sitemap: ...	Point crawlers to your sitemap

Important: crawling vs indexing

A crucial nuance: Disallow stops crawlers from fetching a page, but it does not guarantee the page stays out of search results — a blocked URL can still be indexed if other sites link to it. To reliably keep a page out of search, use a noindex meta tag (and let it be crawlable). Use robots.txt to manage crawl budget and block private areas, not as your only privacy measure.

Best practices

• Always include your sitemap so crawlers find all your pages.
• Don't block CSS/JS that pages need to render properly.
• Be careful with "Disallow: /" — it blocks your entire site.
• Place the file at the root — it only works at /robots.txt.

Private and free

The generator runs entirely in your browser — nothing is uploaded. The tool is completely free with no sign-up.