What robots.txt can and cannot do (it cannot hide a page)

robots.txt is a plain text file at the root of your domain that tells well-behaved crawlers where not to go. It is the oldest convention on the web that anyone still uses daily, and it is misunderstood in one specific, expensive way.

The distinction everything hinges on

robots.txt controls crawling. It does not control indexing.

Crawling is a bot fetching the page. Indexing is a search engine deciding to list it in results. They are separate decisions, and blocking the first does not prevent the second.

If several sites link to a URL you have disallowed, Google can index it on the strength of those links alone — showing the URL and the anchor text people used, with a note that no description is available because the page could not be crawled. The page you blocked is in the results, described by strangers.

Worse: because you blocked it, Google cannot see any noindex tag you subsequently add. The block prevents the fetch that would have delivered the instruction. Blocking a page to hide it can make it permanently un-hideable.

So how do you actually hide a page?

• Keep it out of search results → <meta name="robots" content="noindex"> on the page, or the X-Robots-Tag: noindex HTTP header for non-HTML files. Leave it crawlable so the instruction can be read.
• Keep it away from people → password protection or an IP restriction. Not a robots directive of any kind. robots.txt is a request that polite bots honour; it is not access control, and it is publicly readable, so a robots.txt listing /admin-backup/ is an advertisement.
• Save crawl budget on pages you do not care about → this is what Disallow is genuinely for. Infinite filter combinations, internal search results, calendar pages stretching to the year 3000.

The rule that avoids the trap: never Disallow a URL you also want deindexed. Let it be crawled, serve noindex, wait for it to drop out, and only then block it if you want to save the crawl.

Syntax

User-agent: *
Disallow: /admin/
Disallow: /*?sort=
Allow: /admin/public-page

Sitemap: https://example.com/sitemap.xml

Points that catch people out:

• It must be at the root. example.com/robots.txt. Nowhere else is read.
• It is per host and per protocol. www and non-www are different files. So, technically, are http and https.
• Paths are case-sensitive. /Admin/ and /admin/ are different.
• Only one group applies. A crawler uses the single most specific User-agent block that matches it and ignores every other, including the wildcard. A rule you put under User-agent: * does not apply to Googlebot if a User-agent: Googlebot block exists anywhere in the file.
• Allow beats Disallow when it is more specific — that is how you open one path inside a blocked directory.
• Crawl-delay is ignored by Google. Bing honours it. Google's crawl rate is managed in Search Console.
• An empty Disallow: means allow everything. Disallow: / means block everything. One character apart.

The mistakes that cost real traffic

Shipping the staging file. Staging sites carry Disallow: / to keep them out of the index. Deploy that to production and the whole site is blocked. It is the most expensive two-character mistake in web development, it happens regularly, and the symptom — a slow, total disappearance from search — takes weeks to notice.

Blocking CSS and JavaScript. Common advice in 2010, actively harmful now. Google renders pages to judge them; a page whose stylesheet is blocked renders as unstyled text, and mobile-friendliness assessment fails on a page that is perfectly fine.

Expecting Disallow to deindex. Covered above, and the single most common misunderstanding.

Blocking a page you also canonicalise from elsewhere. The canonical cannot be read if the target cannot be crawled.

Assuming everyone obeys it. Legitimate search engines do. Scrapers, spam bots and content harvesters do not, and never have. If a bot is a problem, block it at the server or the firewall.

What to put in it for a normal site

Most sites need very little. Disallow genuinely useless paths — internal search results, cart and checkout, faceted URL parameters that generate infinite combinations, admin — and list your sitemap. That is usually all.

Resist the temptation to block things pre-emptively. A robots.txt file with forty rules on a hundred-page site is a liability: every rule is an opportunity to accidentally block something that matters, and the benefit at that scale is zero.

Our robots.txt generator produces a correct file for the common cases, and the robots.txt tester answers the question that matters — given this file, is this specific URL allowed for this specific crawler — which is much harder to work out by reading than people expect once Allow rules and wildcards are involved.

Sitemaps

The Sitemap: line is worth having. It applies to the whole file regardless of user-agent groups, and it is how a crawler that arrived without going through Search Console finds your URL list.

A sitemap is a suggestion, not a guarantee of indexing, and it should list only URLs you actually want indexed: canonical, 200-returning, not noindexed. A sitemap full of redirects and 404s is a quality signal, and not a good one. The sitemap generator and sitemap validator cover both ends of that.

The one-line summary

robots.txt says "please do not go here". It does not say "please forget you ever heard of this". For that you need noindex — and for noindex to work, the page has to be crawlable.