In order to configure robots.txt for the Realurl extension in TYPO3 you need to set two things:
- Add filename for page type 201 in realurl_config.php
- Add some TypoScript to process robots generation
Add filename for page type in realurl_config.php
$TYPO3_CONF_VARS['EXTCONF']['realurl']['_DEFAULT'] = array( ... // configure filenames for different pagetypes 'fileName' => array( 'defaultToHTMLsuffixOnPrev' => 0, 'index' => array( 'print.html' => array( 'keyValues' => array( 'type' => 98, ), ), ... // add robots.txt page type 'robots.txt' => array( 'keyValues' => array( 'type' => 201 ) ) ), ...
Add TypoScript to process robots.txt generation
robots = PAGE robots { typeNum = 201 config { disableAllHeaderCode = 1 additionalHeaders = Content-type:text/plain } 10 = TEXT 10 { wrap ( User-Agent: * Allow: / #allow everything by default Disallow: /fileadmin/templates #then add restrictions Disallow: /typo3/ Disallow: /t3lib/ Disallow: /typo3conf/ Disallow: /typo3temp/ Disallow: /*?id=* User-agent: googlebot # Google specific settings Disallow: /*?tx_indexedsearch Sitemap: /?eID=dd_googlesitemap # finally add some sitemap ) }
Be sure to flush the cache and you are all set!
Brilliant solution !! thank you very much for sharing …
I have one problem … the robots.txt is generated not respecting the multiline, it just writes it all in one line, I couldn’t figure out how to correct this and I fear this is incorrect syntax … (I write 07/2017)
I use the script on another site (one of the multisite is http://www.lagottotartufo.it)