Robots Txt Disallow All

Learn how to block unwanted bots from your website with Robots.txt Disallow All. Example: Disallow: /wp-admin/*

What is a Robots.txt Disallow All Rule?

A Robots.txt Disallow All rule is a command used in the robots.txt file that tells web robots to not crawl and index any pages on a website. This rule can be used to prevent web robots from accessing parts of a website that you don't want to be indexed, such as pages with sensitive information.

The Robots.txt Disallow All rule is typically used in conjunction with other rules in the robots.txt file, such as allowing certain web robots to crawl certain pages on the website. For example, if you wanted to prevent all web robots from crawling a site, but still allow Googlebot to crawl the home page, you would use the Robots.txt Disallow All rule and then provide a separate rule allowing Googlebot to access the home page.

Example of Robots.txt Disallow All Rule

The following is an example of a Robots.txt Disallow All rule. This example will prevent all web robots from crawling any pages on the website.

User-agent: *
Disallow: /

In this example, the first line specifies that the rule applies to all web robots (indicated by the asterisk). The second line specifies the rule - in this case, the Disallow rule, which tells all web robots to not crawl any pages on the website.