Tag: 201

In order to configure robots.txt for the Realurl extension in TYPO3 you need to set two things:

1. Add filename for page type 201 in realurl_config.php
2. Add some TypoScript to process robots generation

Add filename for page type in realurl_config.php

$TYPO3_CONF_VARS['EXTCONF']['realurl']['_DEFAULT'] = array(
  ...
  // configure filenames for different pagetypes
    'fileName' => array(
        'defaultToHTMLsuffixOnPrev' => 0,
        'index' => array(
            'print.html' => array(
                'keyValues' => array(
                    'type' => 98,
                ),
            ),
            ...
            // add robots.txt page type
            'robots.txt' => array(
                'keyValues' => array(
                    'type' => 201
                )
            )
        ),
  ...

Add TypoScript to process robots.txt generation

robots = PAGE
robots {
  typeNum = 201
  config {
    disableAllHeaderCode = 1
    additionalHeaders = Content-type:text/plain
  }
  10 = TEXT
  10 {
    wrap (
User-Agent: *
Allow: /                #allow everything by default
Disallow: /fileadmin/templates     #then add restrictions
Disallow: /typo3/        
Disallow: /t3lib/        
Disallow: /typo3conf/    
Disallow: /typo3temp/    
Disallow: /*?id=*        

User-agent: googlebot  # Google specific settings
Disallow: /*?tx_indexedsearch

Sitemap: /?eID=dd_googlesitemap # finally add some sitemap
    )
} 

Be sure to flush the cache and you are all set!