Configure robots.txt for Realurl in TYPO3

TYPO3 Logo

In order to configure robots.txt for the Realurl extension in TYPO3 you need to set two things:

  1. Add filename for page type 201 in realurl_config.php
  2. Add some TypoScript to process robots generation

Add filename for page type in realurl_config.php

$TYPO3_CONF_VARS['EXTCONF']['realurl']['_DEFAULT'] = array(
  ...
  // configure filenames for different pagetypes
    'fileName' => array(
        'defaultToHTMLsuffixOnPrev' => 0,
        'index' => array(
            'print.html' => array(
                'keyValues' => array(
                    'type' => 98,
                ),
            ),
            ...
            // add robots.txt page type
            'robots.txt' => array(
                'keyValues' => array(
                    'type' => 201
                )
            )
        ),
  ...

Add TypoScript to process robots.txt generation

robots = PAGE
robots {
  typeNum = 201
  config {
    disableAllHeaderCode = 1
    additionalHeaders = Content-type:text/plain
  }
  10 = TEXT
  10 {
    wrap (
User-Agent: *
Allow: /                #allow everything by default
Disallow: /fileadmin/templates     #then add restrictions
Disallow: /typo3/        
Disallow: /t3lib/        
Disallow: /typo3conf/    
Disallow: /typo3temp/    
Disallow: /*?id=*        

User-agent: googlebot  # Google specific settings
Disallow: /*?tx_indexedsearch

Sitemap: /?eID=dd_googlesitemap # finally add some sitemap
    )
} 

Be sure to flush the cache and you are all set!

Comments

One response to “Configure robots.txt for Realurl in TYPO3”

  1. gerry Avatar

    Brilliant solution !! thank you very much for sharing …
    I have one problem … the robots.txt is generated not respecting the multiline, it just writes it all in one line, I couldn’t figure out how to correct this and I fear this is incorrect syntax … (I write 07/2017)

    I use the script on another site (one of the multisite is http://www.lagottotartufo.it)

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.