Robots txt settings

Learn how to configure a robots.txt file, including an example, to control search engine indexing and crawling on your website.

Robots.txt File Basics

Robots.txt is a file used to control how search engine robots crawl and index your website's content. It is a text file placed in the root directory of a website that gives instructions to robots about which pages or folders should be indexed and which should not. The robots.txt file helps search engine bots understand which areas of your website are off-limits and should not be crawled. This can help improve the performance of your website by preventing robots from slowing down the server with unnecessary requests.

An example of a robots.txt file looks like this:

User-agent: *
Disallow: /admin/
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/

The first line in the robots.txt file tells search engine robots which user agent they should follow the instructions for. The asterisk (*) signifies that the instructions should be followed by all robots. The next lines specify which files or folders should not be indexed by the robots. In this example, the admin, cgi-bin, wp-admin, and wp-includes folders are blocked from being indexed.

You can also use robots.txt files to specify the location of your sitemap. By adding the following line to the robots.txt file, you can tell robots where to find your sitemap:

Sitemap: https://www.example.com/sitemap.xml

Robots.txt is an important part of any website. It helps search engine bots understand which parts of your website should be crawled and indexed, and which should not. By using the robots.txt file, you can ensure that your website is properly indexed and that your server performance is not affected by unnecessary requests.