The default version of the robots.txt file has been updated as from Joomla 3.3 to make it more search engine friendly allowing Google to access and crawl the /media/ and /templates/ directories.
The old version of the file was basically stopping Google from crawling the /media/ and /templates/ directories and preventing Google from accessing the CSS, Java and image files.
The updates to the robotos.txt file is not applied automatically after you upgrade to the latest version of Joomla 3.
You will be required to manually make the changes to the file to allow google to crawl and index your website correctly.
Old version of the Robots.txt
As you can see from the screen shot below the old version of the robots.txt file is blocking Google from crawling the /media/ and /templates/ directories.
New version of the Robots.txt file
As you can see in the new updated version of the robots.txt file screen shot Google is allowed to access and crawl the /media/ and /Templates/ Directories.
Why should you update the Robots.txt file
-
In July 2015 Google updated the webmaster guidelines to advise website owners that blocking Googlebot from accessing javascript and CSS files could effect the your website ranking in the search engine listing.
- If Google can not access the the /media/ and /templates/ directories it will not be able to render the website correctly.
- The outdated version of Robotos.txt file will generate issues for you via the Google search engine console and when you test your website via the Google mobile friendly test tool.
What Is the Robots.txt File?
The robots.txt is stored in the root or home directory of your Joomla website and basically tells search engine crawlers which part of your website you want them to crawl and which parts of the website you don’t want them to crawl and index.
For more detailed information read the articles learn more about Robots.txt files and the exclusion standards which can be applied to the file.
How do I update the Joomla Robots.txt file
- Open the FTP Software installed on your computer.
- FTP to the root or home directory of your website.
- Download the robots.txt file to your computer. If you don’t have a robots.txt file you will have a robots.txt.dist file download this file to your computer.
- Open the .txt file using notepad.exe on your computer. If you’re using the robots.txt.dist just remove the .dist from the file name.
- Check the file to ensure the /media/ and /templates/ directory are not included in the file.
- If they are included delete the lines Disallow: /media/ and Disallow: /templates/ and save the changes to the txt file.
- Upload the updated robots.txt file to the root or home directory of your website.
That’s it, the Google search bot can now fully crawl the directories at your website in order to index and render your website correctly.
Once you have made the changes you can log into the Google search engine console and use the robots.txt tester tool to confirm if any directories, files are getting blocked.
