Robots.txt generator
Create a robots.txt file: agents, permissions, sitemap.
- Instant
- Free
- Private (processed locally)
- No sign-up
Guide the bots, with no syntax error
A badly written robots.txt can accidentally block your whole site. This tool assembles a valid file from your choices: CMS presets, paths to exclude, sitemap. You stay in control, with no risk of a typo.
-
Pick a preset
Allow all, block all, or a CMS (WordPress, Shopify, Joomla).
-
Refine the rules
User-agent, disallowed paths, crawl delay, sitemap.
-
Drop the file
Copy the result into a robots.txt file at the site root.
The essential directives
| Directive | Effect |
|---|---|
| User-agent: * | Applies to all bots |
| Disallow: /admin/ | Forbids crawling of /admin/ |
| Disallow: | Forbids nothing (everything allowed) |
| Sitemap: … | Gives the site map URL |
Everything is generated locally, in your browser. Always check your file in Search Console before putting it live on a production site.
Frequently asked questions
What is the robots.txt file for?
Placed at a site’s root (https://example.com/robots.txt), it tells search-engine crawlers which parts to explore or avoid. It’s a courtesy convention: respectful bots follow it, but it doesn’t technically block access.
Is blocking a page enough to hide it from Google?
No. “Disallow” prevents crawling, not indexing: a blocked page can still appear in results if other sites link to it. To truly exclude it, use a “noindex” meta tag or password protection.
What does an empty “Disallow:” mean?
A “Disallow:” line with no value allows everything: the bot may crawl the whole site. Conversely, “Disallow: /” blocks everything. That’s the difference between “nothing forbidden” and “everything forbidden”.
Should I list the sitemap here?
It’s recommended: the “Sitemap:” line gives engines the URL of your site map, which speeds up discovery of your pages. Use a full, absolute URL, e.g. https://example.com/sitemap.xml.