How to prohibit page indexing in Robots Txt

Learn how to prevent search engines from indexing your website pages using a robots.txt file & example.

Indexing Restrictions in Robots.txt

Robots.txt is a text file used to inform search engine crawlers which pages should not be crawled. This helps to control which content is indexed and made available to users searching the web. For example, if you have sensitive or private information that should not be found in search engine results, you can add it to your robots.txt file to prevent it from being indexed.

To prohibit page indexing in robots.txt, you need to add a Disallow directive to the file. This directive tells search engine crawlers which pages should not be included in the index. For example, if you wanted to prevent all pages on your site from being indexed, you could add the following to your robots.txt file:

User-agent: *
Disallow: /

The User-agent: * directive applies the rule to all search engine crawlers, while the Disallow: / directive prevents all pages on the site from being indexed. If you only wanted to prevent certain pages from being indexed, you could add a specific path to the Disallow directive. For example, if you wanted to prevent the page /my-private-page from being indexed, you could add the following to your robots.txt file:

User-agent: *
Disallow: /my-private-page

It is important to note that search engine crawlers are not required to obey robots.txt directives. If you have sensitive or private information that should not be indexed, it is best to use other methods such as password protection or noindex tags to ensure that the content is not indexed.