Since I get a lot of requests for a robots.txt file designed for Magento SEO here is a sample to get you started. This Magento robots.txt makes the following assumptions:
- We don’t differentiate between search engines, hence User-agent: *
- We allow assets to be crawled
- We only allow SEF URLs set in Magento
- e.g. no direct access to the front controller index.php, view categories and products by ID, etc.
- We don’t allow filter URLs
- Please note: The list provided is not complete. In case you have custom extension that use filtering make sure to include these filter URLs and parameters in the filter URLs section.
- We don’t allow session related URL segments
- e.g. product comparison, customer, etc.
- We don’t allow specific files to be crawled
- e.g. READMEs, cron related files, etc.
Enough of the talking, here comes your SEO Magento robots.txt:
# Crawlers Setup
# Paths (clean URLs)
# Misc. files you don’t want search engines to crawl
# Disallow filter urls
Feel free to leave comments below for additional remarks and suggestions for improvement.
4 thoughts on “Sample SEO Magento robots.txt file”
Is this robots.txt file configured for Magento 1.9 or 2.0/2.1? I didn’t see this displayed in the post so wanted toc check…
We just migrated our website to Magento 2.0 and the default Magento 2.0 robot file doesn’t seem complete and doesn’t include the majority of what you’ve posted here.
Thanks for sharing this!
Hi we are trishlaemart.com
Default Robots – Index, flow
i didnt add anything in Miscellaneous Scripts
there is no robots.txt file how to add it any scripts do i need to add its a ecommerce website trishlaemart.com
How to dis allow
Googlebot blocked by robots.txt
Your post will be beneficial for all who are looking for setting up robots.xt for their Magento stores.
However, I have seen one more guide on this same topic: https://magenticians.com/configure-magento-robots-txt/
And found that one step is missing in your post, referring sitemap.
Lastly, thanks for sharing this guide and continue to share more to help people like me 🙂