Sample SEO Magento robots.txt file

Since I get a lot of requests for a robots.txt file designed for Magento SEO┬áhere is a sample to get you started. This Magento robots.txt makes the following assumptions: We don’t differentiate between search engines, hence User-agent: * We allow assets to be crawled i.e. images, CSS and JavaScript files...