✅ Correct — this page is genuinely noindexed

This page is noindexed via an HTTP response header: X-Robots-Tag: noindex. There is no <meta name="robots"> tag in the HTML — search for it in view-source and you won't find one. The noindex is only visible in the response headers. To check, open Command Prompt (Windows) or Terminal (Mac) and run: curl -I https://sallymills.com/indexing/x-robots-tag/

What this demonstrates

This page carries a noindex directive exclusively in its HTTP response headers — X-Robots-Tag: noindex — with no corresponding meta tag in the HTML. If you open view-source, you will not find a <meta name="robots"> tag anywhere in the <head>.

The directive is set via an Apache .htaccess rule using mod_headers. Googlebot reads response headers before processing HTML, so this method is reliable — but it's invisible to most of the standard view-source checks SEOs use, which is why it's commonly missed in audits.

Why it matters

Most SEO audits check for noindex by looking at the HTML source. X-Robots-Tag is invisible there. If a page has been noindexed via HTTP header and the auditor only checks view-source or the Screaming Frog "Meta Robots" column without also checking response headers, the noindex goes undetected. Pages end up treated as indexable when they're not — or vice versa, when someone adds X-Robots-Tag headers without realising the SEO implication.

X-Robots-Tag is also the only way to noindex non-HTML resources — PDFs, images, video files — that don't have an HTML <head>. In that context it's the correct and only tool. For HTML pages, the meta robots tag in the <head> is usually simpler and more visible. Understanding both methods is important for a thorough audit.

The code

The .htaccess rule that sets the X-Robots-Tag header for this page — and what the HTTP response looks like.

# .htaccess — X-Robots-Tag for this specific file # Requires mod_headers (enabled by default on most Apache installs) <Files "x-robots-tag.html"> Header set X-Robots-Tag "noindex" </Files> # HTTP response headers (from: curl -I https://sallymills.com/indexing/x-robots-tag/) HTTP/1.1 200 OK X-Robots-Tag: noindex Content-Type: text/html # view-source: NO <meta name="robots"> tag visible in HTML # The noindex is ONLY in the response header above

What Google does

  1. Googlebot requests this page and receives the HTTP response.
  2. Before reading the HTML body, Googlebot checks the response headers.
  3. It finds X-Robots-Tag: noindex in the headers.
  4. The noindex directive is applied. Googlebot excludes this page from the index.
  5. There is no meta robots tag in the HTML — the header alone is sufficient.
  6. In Google Search Console, this page appears under "Excluded by 'noindex' tag" — the same category as a meta robots noindex, because the effect is identical.

How to detect it

  • view-source Ctrl+U (Windows) / Cmd+U (Mac) → search for robots in the HTML source. You will find no <meta name="robots"> tag. That absence is the diagnostic clue — the noindex is in the header, not the HTML.
  • curl Open Command Prompt (Windows) or Terminal (Mac) and run: curl -I https://sallymills.com/indexing/x-robots-tag/ → Look for x-robots-tag: noindex in the output. The -I flag fetches response headers only (no page body downloaded) — the only way to see this directive without a tool like Screaming Frog.
  • Google Search Console Will show this page under "Excluded by 'noindex' tag" in the Indexing report — same category as meta robots noindex, since the effect is identical to Google regardless of which method was used.
  • Screaming Frog Response Headers tab → X-Robots-Tag column shows "noindex". Note that the Meta Robots column will be empty or "index" — because there's no meta tag in the HTML. Checking only the Meta Robots column will miss this. Always check response headers as well, especially after migrations.

How to fix it

This page is intentionally noindexed via header — there's nothing to fix here. To make a page indexable that is currently blocked by X-Robots-Tag, remove or comment out the relevant Header set X-Robots-Tag directive in .htaccess (or in your web server config if you're using Nginx), redeploy, and allow Google time to recrawl.

To prevent this issue in future audits: always check response headers alongside HTML source. Include the Response Headers tab in your Screaming Frog crawl configuration and filter for X-Robots-Tag values. Any unexpected noindex values there need investigation.

← Back to Indexing Directives