Robots Txt for the site

Learn how to use robots.txt to control how search engines crawl and index your website, plus get an example for your own site.

Robots.txt File Example

Robots.txt is a text file that lives on your website and tells web crawlers (like search engine bots) which pages of your website to index and which to ignore. The file is very important in ensuring that your website is well indexed and ranked by search engines.

Below is an example of a robots.txt file that can be used for a website:

User-agent: *

Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/

Allow: /wp-content/uploads/

Sitemap: http://example.com/sitemap.xml

In this example, the robots.txt file tells all web crawlers not to index any files or directories located in the wp-admin, wp-includes, or wp-content directories. This will help keep your website from being indexed by search engines, as these directories typically contain sensitive information. Additionally, it specifies that web crawlers should index files found in the wp-content/uploads directory, allowing images and other media to be indexed.

Lastly, the robots.txt file specifies the location of the website's sitemap.xml file, which provides a list of all pages and posts that should be indexed on the website. By providing the location of the sitemap, web crawlers can more easily find and index the content they need.