When to use noindex and what robots.txt should say

What noindex and robots.txt are

A noindex tag is a single line in the head of a page that reads <meta name="robots" content="noindex">. Google reads the page, sees the tag and leaves the page out of its results. The page still works for anyone who has the link. On a WordPress site the SEO plugin adds this tag when you switch off the option that says show in search results, either on one page or for a whole type of content.

The robots.txt file is a text file at the root of your site, at your domain followed by /robots.txt, that lists which parts of the site crawlers are asked not to visit and which crawlers are asked to stay out altogether, including the AI assistant crawlers. It stops the crawler before it reads anything, which makes it the wrong tool for hiding a page that is already in Google, since Google will keep the address in its index from links alone and never see a noindex tag it cannot reach.

A local business site needs noindex often and robots.txt almost never.

What a noindex mistake looks like on a home builder site

Ridgeline Homes builds in Kuna and is selling lots from $410,000 in a community called Sagebrush Ridge. The site was built on a staging address in the spring, and the developer checked the WordPress box that reads discourage search engines from indexing this site so that the half-finished pages would stay private. The site went live in June. The box stayed checked.

That one setting did two things. It wrote a blanket rule into robots.txt asking every crawler to stay out of the entire site, and it put a noindex tag on every page. For eleven weeks the builder ran paid ads to a site Google was not listing. A search for new homes Kuna showed two competitors and the county assessor.

The opposite mistake sits on the same site. The community has 14 floor plans, and each plan page has a print view at an address ending in ?print=1, plus filtered views such as ?bedrooms=3 and ?garage=3. None of those carry a noindex tag, so once the blanket setting comes off, Google will index 14 plan pages and about 90 near-copies. The thank-you page that fires the conversion pixel is also indexable and shows up for the builder’s own name.

A worked example: setting noindex and robots.txt for a builder in Kuna

The blanket setting came off first. Then each type of address got a decision, listed in the table.

The robots.txt file went from the staging version to a short one that blocks only what has no content. Here is each version in full.

User-agent: *

Disallow: /

User-agent: *

Disallow: /wp-admin/

Allow: /wp-admin/admin-ajax.php

Disallow: /?s=

Sitemap: https://www.example-builder.com/sitemap_index.xml

The after version keeps crawlers out of the admin folder and the internal search results, allows the one admin file that the site’s forms need, and points to the sitemap. It does not block the print views or the filtered views. Those are handled with noindex and canonical tags, which Google can only read if the crawler is allowed in.

The sold-out phase pages stayed live because buyers who bought in phase one still use them for the plat map and HOA documents. A noindex tag keeps those pages available to anyone with the link while keeping Sagebrush Ridge phase 1 out of search, where it would only bring in buyers for lots that are gone.

How to check your own noindex tags and robots.txt

1. Type your domain followed by /robots.txt into the browser. If you see Disallow: / on its own line under User-agent: *, your whole site is blocked. On WordPress, go to Settings, then Reading, and uncheck the box that says discourage search engines.
2. Open Google Search Console and click Pages under Indexing. In the reasons list, look for Excluded by noindex tag and Blocked by robots.txt. Open each list and confirm every address on it is one you meant to hide. A service page or location page in either list is the problem to fix today.
3. Paste the address of any page into the search bar at the top of Search Console to run URL inspection. The result says whether the page is indexed and, if not, which of the two rules is keeping it out.
4. Search Google for site:yourdomain.com. If the result count is far below the number of pages you have, or zero, one of the two rules is the likely cause.

If the count is far above the number of pages you have, the problem is the reverse one, and the XML sitemap page covers how to trim it.

What we do about noindex and robots.txt

You will get a list of every address on your site sorted into pages that should rank and pages that should not, with a noindex tag on the second group and a robots.txt file that blocks only the folders with no content in them. Any blanket block left over from a staging site is removed the same day it is found. Each page you want to rank is checked in Search Console to confirm it is indexed, and each page you want hidden is checked to confirm it is gone. This check is run on every site before any other work starts.

Questions people ask about SEO: when to use noindex and what robots.txt should say