The robots.txt file is the first thing Googlebot reads when it visits your site. A simple text file, under 1KB, but it decides which pages get crawled and which are ignored. A small mistake in it can hide your entire site from Google.
What Is robots.txt?
It's a plain text file living at the root of your site (https://example.com/robots.txt). It contains rules for search engine crawlers (Googlebot, Bingbot, etc.) telling them: "enter these paths, avoid those."
The standard is called Robots Exclusion Protocol, developed in 1994. Today every respectful search engine (Google, Bing, Yandex, DuckDuckGo) follows it.
The Basic Directives
User-agent — specifies the crawler. * means all crawlers.
Disallow — blocks crawling on a path. Example: Disallow: /admin/.
Allow — overrides a Disallow rule for a specific path. Useful when blocking a folder but allowing a file inside.
Sitemap — tells crawlers where the sitemap is.
Crawl-delay — requests crawlers wait between requests. Google ignores this; Bing respects it.
A Complete robots.txt Example
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /search?
Sitemap: https://example.com/sitemap.xmlDisallow vs. noindex
The biggest mistake beginners make: they think Disallow prevents indexing. It does not.
Disallow blocks crawling but doesn't prevent indexing. If a Disallowed page has external links, Google may index it with title only (no content).
noindex (in meta tag) actually prevents indexing. But for it to work, Googlebot must be able to reach the page and read the meta tag — meaning it must not be blocked by Disallow.
Rule: to fully prevent indexing, use noindex and do NOT use Disallow on the same page.
What Should You Block?
Block: Admin dashboards, API endpoints, internal search pages, dynamic filter pages (with many parameters), empty search results.
Don't block: CSS or JavaScript files. Google needs to read them to understand site design. Blocking them weakens your ranking.
Don't block: Images if you want them to appear in Google Images.
Catastrophic Mistakes
Mistake 1: Disallow: / — blocks the entire site. Very common when moving from dev to production, where developers forget to remove this line.
Mistake 2: Blocking CSS and JS. Google can't see your design and assumes it's broken or outdated.
Mistake 3: Blocking important SEO tools like Ahrefs and Semrush. These tools help analyze your site and aren't harmful.
Mistake 4: Leaving the file empty or non-existent. Google assumes you're allowing everything, which may not be intentional.
Rules for Specific Crawlers
You can write different rules for different crawlers. Example: allow Googlebot everything, block ChatGPT's crawler (GPTBot) from training on your content:
User-agent: *
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /Testing Your robots.txt
Before deploying, test the file. Google Search Console has a robots.txt Tester. It lets you enter a URL and see whether your rules block it or not.
Alternatives: free online tools, or visit a URL with ?googlebot=1 to see it as Google would.
Frequently Asked Questions
Do I need robots.txt if my site is small? Yes. Even if you have no paths to block, the file tells Google where your sitemap is.
How do I update robots.txt? After editing, Google picks up the update within 24-48 hours.
Start Now
Writing robots.txt by hand is error-prone. Use the free robots.txt Generator which comes with presets for WordPress, Shopify, and Next.js, and lets you add multiple rules easily. Then sync with your Sitemap and verify everything via SEO Audit.