Tagged: robots.txt

Magento Logo 4

Sample SEO Magento robots.txt file

Since I get a lot of requests for a robots.txt file designed for Magento SEO┬áhere is a sample to get you started. This Magento robots.txt makes the following assumptions: We don’t differentiate between search engines, hence User-agent: * We allow assets to be crawled i.e. images, CSS and JavaScript files...

TYPO3 Logo 1

Configure robots.txt for Realurl in TYPO3

In order to configure robots.txt for the Realurl extension in TYPO3 you need to set two things: Add filename for page type 201 in realurl_config.php Add some TypoScript to process robots generation Add filename for page type in realurl_config.php Add TypoScript to process robots.txt generation Be sure to flush the...