# What is X-Robots-Tag?

> X-Robots-Tag is an HTTP response header that carries robots directives such as noindex or nosnippet, so they also work for PDFs, images and other files.

URL: https://yes-seo.com/glossary/x-robots-tag

**X-Robots-Tag** is an HTTP response header that carries the same instructions as the robots meta tag — `noindex`, `nofollow`, `nosnippet` and the rest — but at the server level. Because it travels with the response rather than inside the HTML, it works for any file type: PDFs, images, videos, spreadsheets, even plain text.

```
HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex, nofollow
```

## When it beats the meta tag

- **Non-HTML files** have no `<head>` to hold a meta tag. A price list PDF or a scanned contract can only be kept out of results this way.

- **Rules by pattern**: one server rule can noindex every file in `/exports/` or every `.docx` on the site.

- **No template access**: on platforms where you can’t edit the HTML head but can set headers at the CDN or proxy.

## Configuration examples

Apache (virtual host or `.htaccess`, with `mod_headers` enabled):

```

  Header set X-Robots-Tag "noindex"

```

Nginx:

```
location ~* \.(pdf|docx?)$ {
  add_header X-Robots-Tag "noindex" always;
}
```

You can target a single crawler by prefixing its name — `X-Robots-Tag: googlebot: nosnippet` — and send the header more than once. Google lists every supported rule, including `max-snippet`, `max-image-preview` and `unavailable_after`, in its [robots meta tag and X-Robots-Tag specification](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag).

## Removing a file that’s already indexed

Add the header, keep the file crawlable, and wait for Google to fetch it again. For a PDF that rarely gets recrawled, request inspection of its URL in Search Console. If the file should disappear entirely, deleting it and returning 404 or 410 works as well.

## How conflicts resolve

When the header and a meta tag disagree, Google applies the most restrictive combination. A page sending `X-Robots-Tag: noindex` stays out of the index even if its HTML says `index, follow`.

## Common mistakes

- **A staging rule shipped to production.** One `add_header X-Robots-Tag noindex` in a shared server block can deindex a whole site, and nobody sees it when viewing the page source.

- **Blocking the file in robots.txt as well**, so crawlers never receive the header — the same trap as with a [noindex](https://yes-seo.com/glossary/noindex) meta tag. See [robots.txt](https://yes-seo.com/glossary/robots-txt).

- **Nginx header inheritance.** Headers from a parent block are dropped as soon as a nested `location` defines its own `add_header`, so your rule can silently vanish.

- **Forgetting the CDN.** Some CDNs and hosting platforms add or strip headers. Check the live response from outside, not just the config file.

## Check your headers

The [HTTP header checker](https://yes-seo.com/http-header-checker) shows the raw response headers for any URL, including X-Robots-Tag, and explains the SEO-relevant ones. To combine that with robots.txt, meta robots and the canonical in a single verdict, use the [indexability checker](https://yes-seo.com/indexability-checker).

## Frequently asked questions

### Is X-Robots-Tag an official web standard?

It isn’t part of an RFC, but it is a long-established convention supported by Google, Bing and other major search engines, with the same directives as the robots meta tag.

### Can I use X-Robots-Tag on normal HTML pages?

Yes. It works on any response, HTML included. Many sites prefer the meta tag for HTML because it’s visible in the source, and use the header for files and pattern-based rules.

### How do I see whether a URL sends an X-Robots-Tag header?

Look at the response headers in your browser’s developer tools (Network tab), run curl -I with the URL, or use an HTTP header checker. Test the public URL, since CDNs can change headers.