Robots.txt Generator
Control what crawlers can access: build rule groups per user-agent, add your sitemap, and copy a valid robots.txt.
robots.txt
User-agent: *
Disallow: /admin/
Disallow: /private/
How to use
- Add a rule group per crawler (or * for all).
- List Disallow and Allow paths, one per line.
- Add your sitemap URL and copy the file to your site root.
Examples
Block admin for everyone
Output: User-agent: * / Disallow: /admin/
Frequently asked questions
Does robots.txt hide pages from Google?
It blocks crawling, not indexing. A linked page can still appear in results without a description. Use noindex to de-index.
Where does the file go?
At the root: https://yourdomain.com/robots.txt.