Configure robots.txt for Realurl in TYPO3

In order to configure robots.txt for the Realurl extension in TYPO3 you need to set two things:

  1. Add filename for page type 201 in realurl_config.php
  2. Add some TypoScript to process robots generation

Add filename for page type in realurl_config.php

$TYPO3_CONF_VARS['EXTCONF']['realurl']['_DEFAULT'] = array(
  ...
  // configure filenames for different pagetypes
    'fileName' => array(
        'defaultToHTMLsuffixOnPrev' => 0,
        'index' => array(
            'print.html' => array(
                'keyValues' => array(
                    'type' => 98,
                ),
            ),
            ...
            // add robots.txt page type
            'robots.txt' => array(
                'keyValues' => array(
                    'type' => 201
                )
            )
        ),
  ...

Add TypoScript to process robots.txt generation

robots = PAGE
robots {
  typeNum = 201
  config {
    disableAllHeaderCode = 1
    additionalHeaders = Content-type:text/plain
  }
  10 = TEXT
  10 {
    wrap (
User-Agent: *
Allow: /                #allow everything by default
Disallow: /fileadmin/templates     #then add restrictions
Disallow: /typo3/        
Disallow: /t3lib/        
Disallow: /typo3conf/    
Disallow: /typo3temp/    
Disallow: /*?id=*        

User-agent: googlebot  # Google specific settings
Disallow: /*?tx_indexedsearch

Sitemap: /?eID=dd_googlesitemap # finally add some sitemap
    )
} 

Be sure to flush the cache and you are all set!

1 thought on “Configure robots.txt for Realurl in TYPO3”

  1. Brilliant solution !! thank you very much for sharing …
    I have one problem … the robots.txt is generated not respecting the multiline, it just writes it all in one line, I couldn’t figure out how to correct this and I fear this is incorrect syntax … (I write 07/2017)

    I use the script on another site (one of the multisite is http://www.lagottotartufo.it)

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Scroll to Top