Meta robots
The meta robots tag is an HTML element placed in the <head> of a document that gives search engine crawlers instructions about that specific page: whether it should be indexed and whether the crawler should follow the links it contains. It is one of the most important tools for precisely controlling indexing in SEO.
How the meta robots tag works
The tag takes one or more directives in its content attribute. The most important are: index / noindex (whether the page should appear in search results) and follow / nofollow (whether the crawler should follow the links and pass authority to them). By default, when the tag is absent, crawlers assume index, follow. Additional values are available, such as noarchive (no cached copy), nosnippet or max-snippet, which control how the page is presented in the SERP. The same directives can also be sent via the X-Robots-Tag HTTP header, which is useful for non-HTML files such as PDFs.
Practical application
The meta robots tag is used to keep pages with no search value out of the index: internal search result pages, the cart, login pages, print versions, form thank-you pages and duplicate variants. The noindex, follow combination lets you hide a page from results while still letting authority flow through its links to the rest of the site.
It is important not to confuse meta robots with the robots.txt file: for a crawler to read a noindex directive, it must be able to reach the page, so blocking the same path in robots.txt means the tag will never be read. Alongside meta robots, the canonical tag is often used to indicate the preferred version of a page.
Powiązane pojęcia
Najczęstsze pytania
How is the meta robots tag different from robots.txt?
The robots.txt file controls whether a crawler is allowed to fetch a given path and works at the whole-server level. The meta robots tag works at the level of a single page and controls its indexing and link following — but the crawler must first be able to reach the page to read it. That is why a page blocked in robots.txt cannot be reliably marked noindex.