Robots.txt Generator
Create robots.txt with User-Agent, Allow, Disallow, Crawl-Delay rules and Sitemap reference. Validate rule correctness.
Rule 1
Allow
Disallow
robots.txt
User-agent: *
Disallow: /admin/
Disallow: /api/
Sitemap: https://example.com/sitemap.xmlFrequently Asked Questions
Where should robots.txt go?
At the site root: https://example.com/robots.txt
Does Disallow prevent indexing?
No. Disallow blocks crawling. To prevent indexing use meta robots noindex.
Difference between Allow and Disallow?
Disallow blocks crawling; Allow exempts a path from a Disallow rule.
How do I add a Sitemap?
On its own line: Sitemap: https://example.com/sitemap.xml