Does blocking a page in robots.txt remove it from Google Search? No. Google says a blocked URL can still appear in results if it is discovered elsewhere. To keep an accessible page out of Google's index, use a supported noindex directive and allow Google to crawl it so the directive can be read. Use authentication when the content itself must be private.

A small team can easily confuse three decisions: whether a visitor may access a page, whether a crawler may fetch it, and whether a search engine may index it. They need different controls. This guide focuses on Google Search, not a promise about every search engine or AI product.

Robots.txt controls crawling

Google's introduction to robots.txt explains that robots rules manage crawler access. They are not a reliable way to keep a web page out of search results. A URL blocked from crawling may still be indexed without its content when Google finds links to it elsewhere.

For a fictional example, imagine a public campaign page that has been linked from a partner's website. Adding its path to robots.txt does not make that address unknown. The team should first decide whether the page needs to remain accessible, disappear from search, or become private.

Noindex needs to be visible to the crawler

Google's noindex documentation describes a robots meta tag for HTML and an X-Robots-Tag response header. It also warns that if robots.txt blocks crawling, Google cannot see a noindex directive on that page. Google does not support putting noindex directly in robots.txt.

That creates an important implementation check: do not combine a crawl block with a page directive and assume the second control has been processed. Inspect the actual response at the exact URL, including redirects. A directive in a template that is not used by the live page does nothing for that response.

For PDFs and other non-HTML files, check the HTTP header rather than looking for an HTML meta element. Ask the person maintaining the site which layer sets the header and whether a CDN or route changes it.

Search exclusion is not access control

If a page contains private customer material, an internal report or restricted account information, make access control the first decision. A noindex instruction does not prevent someone who knows the address from opening a public page. Robots.txt also exposes the paths written in it.

Our suggested decision rule is simple: public and useful pages should have deliberate discovery paths; public pages excluded from search need readable indexing directives; private content needs enforced authorization. This is an operational framework, not a substitute for reviewing your application's security.

Check one URL through the whole path

  1. Name the intended result. Write down whether the page should be publicly accessible, crawlable and indexable. Treat those as three separate answers.
  2. Fetch the real URL. Check its status and redirect destination. Review both HTML robots metadata and response headers.
  3. Read the applicable robots rules. Confirm that the relevant crawler can retrieve the directive you expect it to process.
  4. Inspect internal links and sitemaps. Keep discovery entries consistent with the pages you intend people to find. Removing a sitemap entry alone is not an access restriction or proof of removal.
  5. Check available search evidence. A live response shows current site behavior. A saved indexing report describes the search engine's recorded state. Keep the date and scope with each observation.

These are suggested checks, not results from a customer deployment. Record unresolved outcomes as unknown rather than declaring a page removed because the configuration was saved.

Avoid changing controls just to chase AI visibility

For an article you want readers to discover, an accidental noindex directive can work against that goal. Check the article itself as well as the archive. Conversely, do not make restricted content public merely because someone suggests it will improve GEO.

Keep a copy of the previous configuration before editing. Change only the intended route, verify the live response and then review the recorded search state when available. There is no need to repeatedly submit an unchanged page as a substitute for checking the control.

Our AI discovery guide separates publication, discovery and citation. The AI traffic measurement guide explains why an indexed page, a citation and a recorded visit are different outcomes.

Sources and scope

Official sources checked November 6, 2026. Examples and review steps are Opslumo recommendations. This guide makes no claim about a new launch, guaranteed removal time, AI citation rate or traffic improvement.