Posted on 4 Comments

Sample SEO Magento robots.txt file

Magento Logo

Since I get a lot of requests for a robots.txt file designed for Magento SEO here is a sample to get you started. This Magento robots.txt makes the following assumptions:

  • We don’t differentiate between search engines, hence User-agent: *
  • We allow assets to be crawled
    • i.e. images, CSS and JavaScript files
  • We only allow SEF URLs set in Magento
    • e.g. no direct access to the front controller index.php, view categories and products by ID, etc.
  • We don’t allow filter URLs
    • Please note: The list provided is not complete. In case you have custom extension that use filtering make sure to include these filter URLs and parameters in the filter URLs section.
  • We don’t allow session related URL segments
    • e.g. product comparison, customer, etc.
  • We don’t allow specific files to be crawled
    • e.g. READMEs, cron related files, etc.

Magento robots.txt

Enough of the talking, here comes your SEO Magento robots.txt:

# Crawlers Setup
User-agent: *

# Directories
Disallow: /app/
Disallow: /cgi-bin/
Disallow: /downloader/
Disallow: /includes/
Disallow: /lib/
Disallow: /pkginfo/
Disallow: /report/
Disallow: /shell/
Disallow: /var/

# Paths (clean URLs)
Disallow: /index.php/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /catalogsearch/
#Disallow: /checkout/
Disallow: /control/
Disallow: /contacts/
Disallow: /customer/
Disallow: /customize/
Disallow: /newsletter/
Disallow: /poll/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/
Disallow: /catalog/product/gallery/

# Misc. files you don’t want search engines to crawl
Disallow: /cron.php
Disallow: /cron.sh
Disallow: /composer.json
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /STATUS.txt
Disallow: /mage
#Disallow: /modman
#Disallow: /n98-magerun.phar
Disallow: /scheduler_cron.sh
Disallow: /*.php$

# Disallow filter urls
Disallow: /*?min*
Disallow: /*?max*
Disallow: /*?q*
Disallow: /*?cat*
Disallow: /*?manufacturer_list*
Disallow: /*?tx_indexedsearch

Feel free to leave comments below for additional remarks and suggestions for improvement.

4 thoughts on “Sample SEO Magento robots.txt file

  1. Hi Matthias,

    Is this robots.txt file configured for Magento 1.9 or 2.0/2.1? I didn’t see this displayed in the post so wanted toc check…

    We just migrated our website to Magento 2.0 and the default Magento 2.0 robot file doesn’t seem complete and doesn’t include the majority of what you’ve posted here.

    Thanks for sharing this!

    Gabriel

  2. Hi we are trishlaemart.com

    Default Robots – Index, flow
    i didnt add anything in Miscellaneous Scripts

    there is no robots.txt file how to add it any scripts do i need to add its a ecommerce website trishlaemart.com

  3. How to dis allow
    https://www.googleadservices.com/pagead/conversion.js
    Script
    Googlebot blocked by robots.txt

  4. Your post will be beneficial for all who are looking for setting up robots.xt for their Magento stores.

    However, I have seen one more guide on this same topic: https://magenticians.com/configure-magento-robots-txt/

    And found that one step is missing in your post, referring sitemap.

    Lastly, thanks for sharing this guide and continue to share more to help people like me 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.