Set the redirect with Robots Txt of the old site on Robots Txt new

Learn how to redirect an old website's robots.txt file to a new robots.txt file with an example: Redirect 301 "disallow: /old-directory" "disallow: /new-directory".

Robots Txt Redirect

When migrating a website, it's important to redirect users and search engines to the new location. This is typically done by adding a redirect directive to the old site's Robots.txt file.

For example, if the old website was located at www.example.com and the new website is located at www.example.net, then the following Robots.txt redirect can be added to the old website's Robots.txt file:

User-agent: *
Allow: /
Sitemap: http://www.example.net/sitemap.xml

# Redirect all other requests to the new website
Redirect 301 / http://www.example.net/

The first two lines tell all user-agents (e.g. search engine crawlers) that they are allowed to crawl the old website. The third line points search engines to the new website's sitemap, which is important for indexing the new website. The fourth line contains the actual redirect directive, which tells all user-agents to redirect all requests to the new website.

Once the Robots.txt file is updated with the redirect directive, all user-agents will be redirected to the new website when they attempt to access the old website. This ensures that users and search engines are always accessing the most up-to-date version of the website.