Standard Robots Txt for WordPress

A guide to setting up a standard robots.txt file for WordPress, with an example to get you started.

A robots.txt file is a text file that is located on the root directory of a website. It is used to give instructions to web crawlers on how to index and crawl a website. This is important for WordPress sites since it helps search engines find and index content on the site more efficiently.

Robots.txt for WordPress

The following is an example of a robots.txt file for a WordPress site. This code should be added to the top-level directory of the website:

User-agent: * 
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php 
Allow: /wp-admin/admin.php 
Allow: /wp-admin/async-upload.php 
Allow: /wp-admin/customize.php 
Allow: /wp-admin/edit.php 
Allow: /wp-admin/edit-comments.php 
Allow: /wp-admin/export.php 
Allow: /wp-admin/import.php 
Allow: /wp-admin/index.php 
Allow: /wp-admin/media-new.php 
Allow: /wp-admin/nav-menus.php 
Allow: /wp-admin/options-general.php 
Allow: /wp-admin/plugins.php 
Allow: /wp-admin/post-new.php 
Allow: /wp-admin/profile.php 
Allow: /wp-admin/theme-editor.php 
Allow: /wp-admin/themes.php 
Allow: /wp-admin/tools.php 
Allow: /wp-admin/update-core.php 
Allow: /wp-admin/upload.php 
Allow: /wp-admin/user-new.php 
Allow: /wp-admin/widgets.php 
Allow: /wp-admin/admin-ajax.php 

Disallow: /wp-includes/
Disallow: /wp-content/

# Google Image
User-agent: Googlebot-Image
Allow: /wp-content/uploads/

# Google AdSense 
User-agent: Mediapartners-Google* 
Allow: / 

# MSN 
User-agent: msnbot 
Allow: / 

# Yahoo 
User-agent: Slurp 
Allow: / 

# Ask 
User-agent: Teoma 
Allow: /

# GigaBot 
User-agent: gigabot 
Allow: /

# Yandex 
User-agent: Yandex 
Allow: /

# Allow sitemap
Allow: /sitemap.xml
Allow: /sitemap_index.xml

# Disallow files 
Disallow: /*? 
Disallow: /*.php$
Disallow: /*.js$ 
Disallow: /*.inc$ 
Disallow: /*.css$ 
Disallow: /*.gz$ 
Disallow: /*.rar$ 
Disallow: /*.zip$ 
Disallow: /*.exe$

This robots.txt file will allow web crawlers to access the necessary files on a WordPress site, while blocking access to other files that are not needed for indexing. This will help improve the performance of search engine crawlers on the site, and help improve the overall ranking of the website in search engine results.