Posted on Leave a comment

Setting up Varnish caching server for pd-admin with Apache and php-fcgi

Deploying Varnish caching server can significantly speed up web-based applications, as well as simple websites. Varnish supports different back-end systems, ranging from the popular Apache web server to the more efficient nginx. Especially when it comes to handling high traffic sites Varnish can bring a considerable uplift to the responsiveness by e.g. caching entire pages (i.e. full-page cache).

In order to improve the responsiveness of a couple of high traffic websites running on pd-admin (an advanced collection of tools to administrate web- and mail hosting on Linux-based servers) Varnish 4 was implemented. The following steps describe the process of setting up Varnish caching server for pd-admin with Apache and php-fcgi on a Debian 7 server.

Setup Varnish on Debian 7 64bit

Let’s have a quick look at the process of installing varnish on a Debian 7 64 bit machine. Basically, you need to do 5 things:

  1. Install https support for apt-get
  2. Add GPG key for apt-get
  3. Add the repository to apt-get packages source list
  4. Refresh package source list
  5. Install Varnish

Thus, in order to setup Varnish on Debian 7 64 bit execute the following commands:

  1. apt-get install apt-transport-https
  2. curl https://repo.varnish-cache.org/GPG-key.txt | apt-key add –
  3. echo “deb https://repo.varnish-cache.org/debian/ wheezy varnish-4.0” >> /etc/apt/sources.list.d/varnish-cache.list
  4. apt-get update
  5. apt-get install varnish

That’s it. Varnish should be installed on ready to be configured.

Configure Varnish through VCL – default.vcl

Now that Varnish is installed it’s time to configure it. This can be done by editing default.vcl – the default configuration file automatically created during the installation:

#
# This is an example VCL file for Varnish.
#
# It does not do anything by default, delegating control to the
# builtin VCL. The builtin VCL is called when there is no explicit
# return statement.
#
# See the VCL chapters in the Users Guide at https://www.varnish-cache.org/docs/
# and http://varnish-cache.org/trac/wiki/VCLExamples for more examples.

vcl 4.0;

backend default {
 .host = "127.0.0.1";
 .port = "80";
}

sub vcl_recv {
 # Happens before we check if we have this in cache already.
 #
 # Typically you clean up the request here, removing cookies you don't need,
 # rewriting the request, etc.
}

sub vcl_backend_response {
 # Happens after we have read the response headers from the backend.
 #
 # Here you clean the response headers, removing silly Set-Cookie headers
 # and other mistakes your backend does.
}

sub vcl_deliver {
 # Happens when we have all the pieces we need, and are about to send the
 # response to the client.
 #
 # You can do accounting or modifying the final object here.
}

As starting point have a look at the very handy Varnish 4.0 configuration template which works out of the box for

  • WordPress
  • Drupal (works decently for Drupal 7, depends on your modules obviously)
  • Joomla (WIP)
  • Fork CMS
  • OpenPhoto

and additional configuration setups like

  • Server-side URL rewriting
  • Clean error pages for debugging
  • Virtual Host implementations
  • Various header normalizations
  • Cookie manipulations
  • 301/302 redirects from within Varnish

Have a look at the Varnish configuration documentation to adjust it to your specific needs. In general, only few adaptions should be required using this template.

Set correct backend hostname to solve 403 forbidden error code

In case you get a 403 forbidden error make sure to set the correct backend hostname or IP in default.vcl:

backend server1 { # Define one backend
  .host = "USE_DOMAIN_SET_IN_HTTPD_CONF"; # IP or Hostname of backend
  ...
}

Have a look at pd-admin’s httpd.conf for the currently set hostname to be used by Varnish, or open up the web administration console and check the server name option. Also make sure to have set the X-Forwarded-For header (especially for Varnish 3):


req.http.x-forwarded-for = client.ip

That should solve the 403 forbidden error for cached domains.

Exclude Domains from Varnish Caching

Varnish by default caches all requests that are not excluded specifically. Thus, in case you want to exclude domains and simple URLs from being cached by Varnish you can specify them in vcl_recv() function like so:


if (req.http.host == "www.domain.com" && req.url == "/") {
return (pass);
}

This will redirect specific domains and/or URLs to the pass() function thus by-passing the caching mechanism.

Change Varnish port

Finally, you need to set Varnish to listen on the default http port (i.e. 80) and change Apache’s listener port to something different, e.g. 81 and set it as the Varnish backend port in default.vcl. Make sure to restart Apache and Varnish. That’s it!

Posted on Leave a comment

Disable caching of API callbacks for Viveum Magento extension

Magento Logo

Using a caching server like Varnish is mandatory for running Magento (efficiently). When using extensions that make use of callback API calls make sure that you exclude them from your caching rules.

As we are using Viveum as payment gateway service for most eCommerce projects we generally deploy the official Viveum Magento extension. When using this extension in combination with Varnish make sure that at least exclude the following API calls from your caching list, e.g. through Turpentine URL Blacklist:

ops/payment/
ops/api/

We experienced problems especially related to Paypal transactions without excluding these callbacks, including wrong customer information on the Paypal checkout page.

Posted on Leave a comment

Magento Error ESI processing not enabled

Magento Logo

In a recent Magento setup with a Varnish caching server an undefined PHP variable in a template resulted in the error message “ESI processing not enabled“.

Due to the undefined PHP variable the underlying Varnish caching server’s Edge-side includes (ESI) capabilities, allowing content assembly by HTTP surrogates through in-markup XML-based language failed. After a quick code review of a just installed Magento extension there existed a newly introduced undefined PHP variable that caused Varnish to fail.

Thus, when using Varnish double check for undefined variables in Magento code 😉

Posted on 3 Comments

Setup Varnish Cache Server for Magento 1.9 CE on Mac OSX

Magento Logo

When dealing with high traffic Magento installations you definitely want to implement caching mechanisms to ensure proper response times for your visitors. Apart from internal caching and indexing functionality provided by Magento there exist a couple of (external) extensions that help you speed of content delivery by e.g. caching entire pages, such as PageCache for Varnish.

Simply put, Varnish represents a reverse-proxy cache that serves as entry-point into you Magento installation and returns cached content rather than requesting Magento to re-build entire pages.

The following steps show you can setup Varnish cache server for Magento 1.9 CE on Mac OSX.

First off, make sure you have Homebrew installed and configured properly. Based on Homebrew we can easily setup Varnish by issuing the following command:

bash-3.2$ brew install varnish
==> Installing varnish dependency: pkg-config
==> Downloading http://pkgconfig.freedesktop.org/releases/pkg-config-0.28.tar.gz
######################################################################## 100,0%
==> ./configure --prefix=/usr/local/Cellar/pkg-config/0.28 --disable-host-tool --with-internal-glib --with-pc-path=/usr/local/lib/p
==> make
==> make check
==> make install
  /usr/local/Cellar/pkg-config/0.28: 10 files, 600K, built in 97 seconds
==> Installing varnish dependency: pcre
==> Downloading ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-8.33.tar.bz2
######################################################################## 100,0%
==> ./configure --prefix=/usr/local/Cellar/pcre/8.33 --enable-utf8 --enable-unicode-properties --enable-pcregrep-libz --enable-pcregrep-libbz2 --
==> make
==> make test
==> make install
   /usr/local/Cellar/pcre/8.33: 140 files, 4,1M, built in 49 seconds
==> Installing varnish
==> Downloading http://repo.varnish-cache.org/source/varnish-3.0.3.tar.gz
######################################################################## 100,0%
==> ./configure --prefix=/usr/local/Cellar/varnish/3.0.3 --localstatedir=/usr/local/var
==> make install
==> Caveats
To have launchd start varnish at login:
    ln -sfv /usr/local/opt/varnish/*.plist ~/Library/LaunchAgents
Then to load varnish now:
    launchctl load ~/Library/LaunchAgents/homebrew.mxcl.varnish.plist
Warning: /usr/local/sbin is not in your PATH
You can amend this by altering your ~/.bashrc file
==> Summary
 /usr/local/Cellar/varnish/3.0.3: 51 files, 1,8M, built in 43 seconds

As shown in Brew’s output you can start Varnish by issuing the following command:

==> Caveats
To have launchd start varnish at login:
    ln -sfv /usr/local/opt/varnish/*.plist ~/Library/LaunchAgents
Then to load varnish now:
    launchctl load ~/Library/LaunchAgents/homebrew.mxcl.varnish.plist

In case the following error occurs:

bash-3.2$ sudo launchctl load ~/Library/LaunchAgents/homebrew.mxcl.varnish.plist"
launchctl: Couldn't" stat("~/Library/LaunchAgents/homebrew.mxcl.varnish.plist"): No such file or directory nothing found to load

try a re-install first (and additionally do a brew-update/updgrade):

bash-3.2$ sudo brew uninstall varnish
Uninstalling /usr/local/Cellar/varnish/3.0.3...

After a brew update/upgrade Varnish 4.0.0 is now being installed:

bash-3.2$ sudo brew install varnish
==> Downloading https://downloads.sf.net/project/machomebrew/Bottles/varnish-4.0.0.mavericks.bottle.tar.gz
######################################################################## 100,0%
==> Pouring varnish-4.0.0.mavericks.bottle.tar.gz
==> Caveats
To have launchd start varnish at login:
    ln -sfv /usr/local/opt/varnish/*.plist ~/Library/LaunchAgents
Then to load varnish now:
    launchctl load ~/Library/LaunchAgents/homebrew.mxcl.varnish.plist
==> Summary
 /usr/local/Cellar/varnish/4.0.0: 93 files, 1,9M

Unfortunately, this does not fix the missing plist file problem every time.

Nevertheless, you can easily create the plist file yourself, as mentioned here https://github.com/Homebrew/homebrew/issues/15922:

Simply copy the contents for the plist file by calling brew cat varnish:

bash-3.2$ sudo brew cat varnish
require 'formula'

class Varnish < Formula
  homepage 'http://www.varnish-cache.org/'
  url 'https://repo.varnish-cache.org/source/varnish-4.0.0.tar.gz'
  sha1 '6ad4fcf42a505a748ae993cb6ed579159e023633'

  bottle do
    sha1 "190bba2cf0e521b489cbd64cc698ba769b18cccd" => :mavericks
    sha1 "fd12d468c2328179bd402f1592b65ef69d6f1baa" => :mountain_lion
    sha1 "0e9717e705541a34906947eb7e49ab800204e2ff" => :lion
  end

  depends_on 'pkg-config' => :build
  depends_on 'pcre'

  resource "docutils" do
    url "https://pypi.python.org/packages/source/d/docutils/docutils-0.11.tar.gz"
    sha1 "3894ebcbcbf8aa54ce7c3d2c8f05460544912d67"
  end

  def install
    ENV.prepend_create_path "PYTHONPATH", buildpath+"lib/python2.7/site-packages"
    resource("docutils").stage do
      system "python", "setup.py", "install", "--prefix=#{buildpath}"
    end

    system "./configure", "--disable-dependency-tracking",
                          "--prefix=#{prefix}",
                          "--localstatedir=#{var}",
                          "--with-rst2man=#{buildpath}/bin/rst2man.py",
                          "--with-rst2html=#{buildpath}/bin/rst2html.py"
    system "make install"
    (var+'varnish').mkpath
  end

  test do
    system "#{opt_sbin}/varnishd", "-V"
  end

  def plist; <<-EOS.undent
      <?xml version="1.0" encoding="UTF-8"?>
      <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
      <plist version="1.0">
      <dict>
        <key>Label</key>
        <string>#{plist_name}</string>
        <key>ProgramArguments</key>
        <array>
          <string>#{opt_sbin}/varnishd</string>
          <string>-n</string>
          <string>#{var}/varnish</string>
          <string>-f</string>
          <string>#{etc}/varnish/default.vcl</string>
          <string>-s</string>
          <string>malloc,1G</string>
          <string>-T</string>
          <string>127.0.0.1:2000</string>
          <string>-a</string>
          <string>0.0.0.0:80</string>
        </array>
        <key>KeepAlive</key>
        <true/>
        <key>RunAtLoad</key>
        <true/>
        <key>WorkingDirectory</key>
        <string>#{HOMEBREW_PREFIX}</string>
        <key>StandardErrorPath</key>
        <string>#{var}/varnish/varnish.log</string>
        <key>StandardOutPath</key>
        <string>#{var}/varnish/varnish.log</string>
      </dict>
      </plist>
    EOS
  end
end

The plist content in this case is:

<?xml version="1.0" encoding="UTF-8"?>
      <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
      <plist version="1.0">
      <dict>
        <key>Label</key>
        <string>#{plist_name}</string>
        <key>ProgramArguments</key>
        <array>
          <string>#{opt_sbin}/varnishd</string>
          <string>-n</string>
          <string>#{var}/varnish</string>
          <string>-f</string>
          <string>#{etc}/varnish/default.vcl</string>
          <string>-s</string>
          <string>malloc,1G</string>
          <string>-T</string>
          <string>127.0.0.1:2000</string>
          <string>-a</string>
          <string>0.0.0.0:80</string>
        </array>
        <key>KeepAlive</key>
        <true/>
        <key>RunAtLoad</key>
        <true/>
        <key>WorkingDirectory</key>
        <string>#{HOMEBREW_PREFIX}</string>
        <key>StandardErrorPath</key>
        <string>#{var}/varnish/varnish.log</string>
        <key>StandardOutPath</key>
        <string>#{var}/varnish/varnish.log</string>
      </dict>
      </plist>

Now starting Varnish via launchctl succeeds:

bash-3.2$ sudo launchctl load ~/Library/LaunchAgents/homebrew.mxcl.varnish.plist

Time to test the service

Open a browser and navigate to 127.0.0.1:8080 (which is the port Varnish is listening to set by the configuration in the plist file from above) and check out the headers:

Accept-Ranges	bytes, bytes
Age	0
Connection	keep-alive
Content-Language	de
Content-Length	1087
Content-Type	text/html; charset=utf-8
Date	Fri, 30 May 2014 07:31:12 GMT
Server	Apache/2.4.4 (Unix) PHP/5.4.7 OpenSSL/1.0.1e
Vary	accept-language,accept-charset
Via	1.1 varnish
X-Varnish	1661470811

Congratulations, as you can see Varnish is working properly!

The next step is to configure Apache to listen to port 8080 (or any other port apart from 80) and Varnish on port 80.

That’s it 🙂