robots.txt

robots.txt is a text file placed in a domain's root directory (at example.com/robots.txt) that instructs search engine crawlers which parts of a site they may visit and which they should skip. It is part of the Robots Exclusion Protocol — a voluntary standard that reputable bots such as Googlebot respect. It is usually the first file a crawler requests before it begins scanning a site.

How robots.txt works

The file consists of rules assigned to specific crawlers. The key directives are:

• User-agent — names the crawler the following rules apply to (an asterisk means all),
• Disallow — a path the crawler should not fetch,
• Allow — an exception permitting access to a resource inside a blocked directory,
• Sitemap — the address of the XML sitemap.

For example, the rule Disallow: /admin/ tells crawlers not to scan the admin panel. The crucial point is that robots.txt controls crawling, not indexing — the two are different things.

robots.txt in practice

robots.txt is used mainly to conserve crawl budget: you block resources with no value to search engines, such as internal site searches that spawn endless URLs, login panels or technical files. A common and dangerous mistake is trying to hide a page from Google with Disallow alone — a blocked URL can still surface in results if links point to it. To exclude a page from the index, use the robots meta tag with a noindex value, not robots.txt. When launching a new site, always check that the file is not accidentally blocking everything with Disallow: /, a classic reason a site vanishes from Google.