Host Robots Txt Directive

Learn how to use the Directive host robots.txt to control web crawler access & see an example of how to block certain pages.

Robots.txt Directive Example

Robots.txt is a text file stored on a web server that tells search engine crawlers and other robots which pages and files on the website they can access and which they should not. It is a way of controlling how robots, such as Google and Bing, crawl and index your website.

Here is an example of a robots.txt directive:

User-agent: * 
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /images/
Disallow: /category/
Allow: /

This directive tells robots to not crawl and index any pages in the folders /cgi-bin/, /tmp/, /images/, and /category/. The Allow directive is optional, but it allows robots to crawl and index the entire website.

Robots.txt is an important part of website optimization and SEO, as it can help ensure that only the most relevant pages are indexed, and that search engine crawlers do not waste time crawling pages that are not important to your website.