Robots Txt for Modx

Learn how to control web crawler access with Robots.txt for Modx, plus get an example of how to implement it.

Robots.txt for MODX

The robots.txt file is a text file that communicates with web crawlers and other web robots. The robots.txt file is part of the robots exclusion protocol (REP) and is a standard for websites. It tells web robots which pages to crawl and which pages to ignore.

Using a robots.txt file is a great way to control which pages search engines can access, and can help to prevent certain pages from being indexed. It’s also important to keep in mind that robots.txt is not a 100% guarantee that your pages won't be indexed, as some web crawlers may choose to ignore it.

For MODX, the robots.txt file should be placed in the root directory of the site. Here’s an example of a robots.txt file for MODX:

User-agent: *
Disallow: /assets/
Disallow: /manager/
Disallow: /connectors/
Disallow: /core/
Allow: /

This robots.txt file tells all web robots to disallow access to the assets, manager, connectors, and core directories. It also allows access to all other directories. This ensures that only the publicly accessible pages of a site are indexed by search engines.

It’s also important to note that the robots.txt file should be placed in the root directory of the site. If it's placed anywhere else, it won't be recognized by web crawlers. It's also important to make sure the file is named robots.txt and not something else.

Finally, it’s a good idea to validate the robots.txt file to make sure it’s valid and properly formatted. There are a number of tools available that can help validate robots.txt files. This is especially important if any changes are made to the file, as invalid or improperly formatted files can cause errors.

In conclusion, robots.txt is a great way to control which pages search engines can access and can help to prevent certain pages from being indexed. For MODX, the robots.txt file should be placed in the root directory of the site, and should be validated to make sure it’s valid and properly formatted.