Blog

Setting up an Active Directory domain controller with Samba 4 on a Raspberry Pi 3

The Raspberry Pi is a wonderful platform to simplify your daily IT jobs, such as serving as a media centre for your smart-TV, being the central hub for your home automation system or in the case at hand act as an Active Directory (AD) domain controller in a test lab. Obviously, we are talking about the Samba variant of the Active Directory implementation available since version 4 since the original one offered by the folks at Microsoft requires a x86 architecture which the Raspberry fails to provide using its ARM system. But hey, in the end for this scenario we don’t care too much about the underyling hardware but merely focus on the functional aspect. So let’s begin, shall we?

Raspberry Pi setup used

For the following guide I’ve used a vanilla Raspberry Pi 3 in the following configuration (although the setup should be just fine for other versions too):

$ cat /etc/os-release 
PRETTY_NAME="Raspbian GNU/Linux 9 (stretch)"
NAME="Raspbian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

Our goal at glance

Let’s break down what we need to do in order to achieve our goal to set up an Active Directory domain controller with Samba 4 on a Raspberry Pi 3:

  1. Initial setup of the Raspberry Pi using Raspbian
  2. Setup networking to use a static IP
  3. Install required packages
  4. Disable masked legacy service units
  5. Provision the AD domain
  6. Setup and start required Samba AD domain controller services
  7. Reboot
  8. Check setup by creating new AD user and add a client computer

For this particular setup we are going to use the following base information:

  • router has IP 192.168.1.1
  • hostname is pidc
  • pidc has IP 192.168.1.2
  • domain to be used my.domain.local

Initial setup of the Raspberry Pi using Raspbian

We are not going to cover this here since there are plenty of readups out there. Thus, please check them out and bootstrap your Raspberry Pi using Raspbian. Also make sure that your base system is at the latest version before proceeding further (i.e. sudo apt-get update && apt-get upgrade -V)

Setup networking to use a static IP

With an AD in place you will always want to have a static IP to keep things simple:

$ sudo nano /etc/dhcpcd.conf
# explicitely use eth0 and set static IPs, as well as domain specifics
interface eth0
static routers=192.168.1.1
static domain_name_servers=127.0.0.1
static domain_name_servers=192.168.1.1
static ip_address=192.168.1.2
static domain_search=my.domain.local

Install required packages

Next we need to install the required packages:

$ sudo apt-get install samba smbclient winbind krb5-user krb5-config krb5-locales winbind libpam-winbind libnss-winbind

During the setup you will be asked for

  • Default Kerberos realm
  • Kerberos servers
  • Administrative server

Based on our domain setup you need to enter the following data:

  • Default Kerberos realm: MY.DOMAIN.LOCAL
  • Kerberos servers: my.domain.local
  • Administrative server: pidc.my.domain.local

Disable masked legacy service units

In order to prevent error message related to masked legacy service units issue the following commands to stop and then fully disable them:

$ sudo systemctl stop samba-ad-dc.service smbd.service nmbd.service winbind.service
$ sudo systemctl disable samba-ad-dc.service smbd.service nmbd.service winbind.service

Provision the AD domain

Before being able to actually provision our AD domain let’s do a little house keeping round to make our life easier:

# double-check where the samba config file resides
$ smbd -b | grep "CONFIGFILE"
# let's make a backup of the original samba configuration determined above
$ sudo mv /etc/samba/smb.conf /etc/samba/smb.conf.backup
# also, let's remove the original kerberos configuration, as it will be overwritten and edited later on
$ sudo rm /etc/krb5.conf

Having done those steps it’s finally time to provision our AD domain:

$ sudo samba-tool domain provision --use-rfc2307 --interactive

The provisioning process might take a little. When completed let’s handle the Kerberos configuration:

# again, let's make a backup of the original
$ sudo mv /etc/krb5.conf /etc/krb5.conf.backup
# and symlink to /etc/krb5.conf => NOTE: It's not the best to symlink here but it's OK for now
$ sudo ln -sf /var/lib/samba/private/krb5.conf /etc/krb5.conf

Hint: We will come back to /etc/krb5.conf at the very end for potential missing information so hang in there for now.

Setup and start required Samba AD domain controller services

Finally, let’s start setup and start the required Samba AD domain controller services to get things moving:

$ sudo systemctl unmask samba-ad-dc.service
$ sudo systemctl start samba-ad-dc.service
$ sudo systemctl status samba-ad-dc.service
$ sudo systemctl enable samba-ad-dc.service

Time to check if Samba is running correctly:

$ sudo netstat -tulpn | egrep 'smbd|samba'

Also, make sure to set the search domain and your nameservers in /etc/resolv.conf correctly at this point:

$ sudo nano /etc/resolv.conf

search domain.name.net
nameserver 192.168.1.2
nameserver 192.168.1.1

Once completed, mark /etc/resolv.conf as write-protected to save yourself some pain after reboots:

$ sudo chattr +i /etc/resolv.conf

Reboot

Time to reboot your shiny new AD domain controller setup to take effect:

$ sudo reboot now

Check setup by creating new AD user and add a client computer

# do some simple ping tests
$ ping -c3 my.domain.local
$ ping -c3 pidc.my.domain.local
$ ping -c3 pidc

# test DNS configuration
$ host -t A my.domain.local
$ host -t A pidc.my.domain.local
$ host -t SRV _kerberos._udp.my.domain.local
$ host -t SRV _ldap._tcp.my.domain.local

# test kerberos ticketing => Note the upper-case!
$ kinit administrator@MY.DOMAIN.LOCAL
$ klist
# create a (test) domain user
$ sudo samba-tool user create some.user

More detailed information and the commands available for the create domain user call can be found on the Samba Wiki.

Possible trouble shooting tips for DNS / Kerberos / Samba

If you get an error message like “Cannot contact any KDC for realm while getting initial credentials” first check if Kerberos was in fact started correctly and is listening on port 88 (or a custom port that you’ve defined earlier), e.g. using telnet:

$ telnet pidc.my.domain.local 88

If you are not able to connect (e.g. “Connection refused”) make sure that the Samba services including KDC (Kerberos) is in fact set to be started. For this check the [global] section of your /etc/samba/smb.conf:

$ sudo nano /etc/samba/smb.conf

and add the following line if required:

server services = rpc, nbt, wrepl, ldap, cldap, kdc, drepl, winbind, ntp_signd, kcc, dnsupdate, dns, s3fs

Afterwards restart Samba and re-test KDC kinit:

$ sudo systemctl stop samba-ad-dc.service
$ sudo systemctl start samba-ad-dc.service

Re-test kinit:

$ kinit administrator@MY.DOMAIN.LOCAL

In addition, also make sure that you have a working version of your /etc/krb5.conf, especially for the [realms] and [domain_realm] section:

[libdefaults]
default_realm = MY.DOMAIN.LOCAL
dns_lookup_realm = true
dns_lookup_kdc = true
dns_fallback = yes

[realms]
MY.DOMAIN.LOCAL = {
kdc = PIDC.MY.DOMAIN.LOCAL:88
default_domain = MY.DOMAIN.LOCAL
}

[domain_realm]
.MY.DOMAIN.LOCAL = MY.DOMAIN.LOCAL
MY.DOMAIN.LOCAL = MY.DOMAIN.LOCAL

Finally, make sure that Samba itself is fully started, including all of its services, especially after a reboot:

$ samba

Next steps / Where to go from here

As usual, when dealing with an Active Directory setup you should always have a secondary backup domain controller. The steps to do so are pretty straight forward given the guidelines shown here. Simply hook up a second Raspberry Pi and configure it as your secondary domain controller.

If you are running this setup in a test lab you might not need a backup domain controller but as always make sure to back up your Raspberry as an image to have a quick restore point to go to, e.g. using the following command:

$ dd if=/dev/mmcblk0 of=/your-backup-path/YOUR-BACKUP-NAME-$(date +%Y%m%d-%H%M%S).img bs=1MB

That’s all folks! I hope this saves you some time when setting up an Active Directory domain controller with Samba 4 on a Raspberry Pi 3.

Setting up Collabora CODE with NextCloud using Apache reverse proxy on Debian 8 Jessie

Setting up Collabora Online Development Edition (CODE) can be a little tricky. This guide shows the steps needed to get Collabora CODE working using an Apache 2.2 reverse proxy on Debian 8 Jessie.

The steps we are going to have a look at are as follows:

  1. Setup Apache reverse proxy
  2. Setup Collabora CODE based on official Docker image
  3. Install and configure NextCloud Collabora CODE plugin

The basic configuration we are trying to achieve here is:

  1. Setup a secure domain for accessing Collabora from NextCloud
    1. This will be https://office.yourserver.com
    2. I’m assuming that you already have a working SSL certificate for this domain. If not, have a look at Let’s Encrypt in case you want a cheap solution. For Debian 8 Jessie have a look at the certbot guide.
  2. Setup a reverse proxy configuration for this domain that fowards requests to Collabora webservice endpoints
  3. Access Collabora CODE Docker container through Apache reverse proxy which itself exclusively listens an a secure line on Port 9980 (default)
  4. Install the Collabora NextCloud plugin and configure it to access Collabora through our reverse proxy

Setup Collabora CODE based on official Docker image

Collabora can be either installed using a package provided by your distribution or by using the official Docker image collabora/code.

Normally, the Docker container setup should be pretty pain free. Having said that, for Debian 8 Jessie you need to adjust the storage driver to devicemapper as it seems that the default docker storage driver AUFS and Debian 8 do not work together.

Adjust Docker storage driver to devicemapper

The steps required are again pretty straight forward. First get current ExecStart from your docker.service file:

grep ExecStart /lib/systemd/system/docker.service

Example output:
ExecStart=/usr/bin/dockerd -H fd://

Then use this result to create a systemd Docker drop-in configuration file and create the service directory first if it does not yet exist as well:

mkdir /etc/systemd/system/docker.service.d
editor /etc/systemd/system/docker.service.d/execWithDeviceMapper.conf

Put the following content in execWithDeviceMapper.conf:

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd --storage-driver=devicemapper -H fd://.

Finally, restart systemd, docker.service and possibly your existing Collabora container if you had one running:

systemctl daemon-reload
systemctl restart docker.service

Disclaimer: For higher volume production sites you definitely want to optimize this setup (…).

The command the start your Collabora Docker container is as follows:

docker run -t -d -p 127.0.0.1:9980:9980 \
-e 'domain=www\.yournextcloud1\.com\|www\.yournextcloud2\.com' \
--restart always --cap-add MKNOD collabora/code

Note that I’ve provided two domains in the above command to show how to enable multiple domains to access your Collabora web service.

As always, since the Docker container starts in detached mode make sure to check for possible problems using

docker logs YOUR_CONTAINER_ID

Now that we have Collabora CODE up and running as Docker container we need to make it available to the outside world using an Apache reverse proxy.

Setup Apache reverse proxy

First and foremost, I will not cover the exact steps to setup the base Apache web server here but provide a working vhost configuration.

Required Apache modules

The additional Apache module requirements to get Apache working as reverse proxy for Collabora CODE are:

  1. mod_proxy

  2. mod_proxy_http

  3. mod_proxy_wstunnel

  4. mod_ssl

Apart from mod_proxy_wstunnel the configuration steps should be pretty straight forward. When using Apache 2.2 and mod_proxy_wstunnel on the other hand things can get a little more tricky since you need to apply a patch and compile the module yourself. Have a look at the very handy guide by waleedsamy on github to compile mod_proxy_wstunnel yourself.

Apache Reverse Proxy vhost configuration

Once all requirements are satisfied we can setup the vhost configuration for the Apache reverse proxy domain.

Remember, our internet-facing domain for accessing Collabora CODE will be office.yourserver.com. This will be the basis for your vhost configuration below.

UseCanonicalName off
ServerName office.yourserver.com

# Enable and configure SSL/TLS
SSLEngine on
SSLCertificateFile yourserver-cert
SSLCertificateKeyFile yourserver-key
SSLCertificateChainFile yourserver-cacert

SSLProtocol all -SSLv2 -SSLv3
SSLCipherSuite ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES25$
SSLHonorCipherOrder on
SetEnvIf User-Agent ".*MSIE.*" \
nokeepalive ssl-unclean-shutdown \
downgrade-1.0 force-response-1.0

# Encoded slashes need to be allowed
AllowEncodedSlashes NoDecode

# Enable and configure SSL Proxy
SSLProxyEngine On
SSLProxyVerify None
SSLProxyCheckPeerCN Off

# Make sure to keep the host
ProxyPreserveHost On

# static html, js, images, etc. served from loolwsd
# loleaflet is the client part of LibreOffice Online
ProxyPass /loleaflet https://127.0.0.1:9980/loleaflet retry=0
ProxyPassReverse /loleaflet https://127.0.0.1:9980/loleaflet

# WOPI discovery URL
ProxyPass /hosting/discovery https://127.0.0.1:9980/hosting/discovery retry=0
ProxyPassReverse /hosting/discovery https://127.0.0.1:9980/hosting/discovery

# Main websocket
ProxyPassMatch "/lool/(.*)/ws$" wss://127.0.0.1:9980/lool/$1/ws nocanon

# Admin Console websocket
ProxyPass /lool/adminws wss://127.0.0.1:9980/lool/adminws

# Download as, Fullscreen presentation and Image upload operations
ProxyPass /lool https://127.0.0.1:9980/lool
ProxyPassReverse /lool https://127.0.0.1:9980/lool

Check if your reverse proxy is working by accessing the WOPI discovery URL:

https://office.yourserver.com/hosting/discovery

If that gives you the corresponding XML namespace information you should be good to go.

Install and configure NextCloud Collabora CODE plugin

This is the last step required and should be the easiest one.

  1. Go to the Apps section and choose “Office & Text”
  2. Install the “Collabora Online” app
  3. In Admin -> Collabora Online specific the server you have setup before (https://office.yourserver.com)

Finally, try to create and edit a document via NextCloud. Enjoy your private Collabora setup using NextCloud!

For more information have a look at the official Collabora CODE documentation.

Setting up a classic AWS EC2 Load Balancer for SSL termination

Setting up an AWS EC2 Load Balancer for SSL termination is pretty straight forward once you have all required information at hand.

In this example we will be setting up a classic AWS EC2 Load Balancer for SSL termination.

Before we start configuring the load balancer make sure you have the following required assets at hand:

  1. Server SSL private key
  2. Server SSL certificate
  3. Server SSL certificate chain (if required by your SSL setup)
  4. AWS CLI working
    1. Access to IAM via CLI
    2. AWS Access Key ID
    3. AWS Secret Access Key
  5. Access to the AWS Management Console to setup classic EC2 Load Balancer
  6. Optional: a cup of coffee or tea 😉

We will now go through all the steps required to get your EC2 Load Balancer for SSL termination up and running in no time.

Converting SSL certificates to PEM format

First and foremost, your Server SSL private key and your Server SSL certificate are required to be in PEM format.

If this is not the case use the following commands to convert your private key and certificate to PEM format:

Convert SSL private key to PEM format

openssl rsa -in server-key.key -text > server-key.pem

Convert SSL certificate to PEM format

openssl x509 -inform PEM -in server-certificate.crt > server-certificate.pem

Setup AWS CLI

In order to set up AWS CLI you may use the bundle provided by Amazon or install the requirements manually.

After the installation you will need to setup your AWS CLI credentials.

Make sure that the Policy IAMFullAccess is set for the target AWS CLI user:

Then, setup AWS CLI credentials using the following command:

aws configure

You will be prompted to enter your

  1. AWS Access Key ID

  2. AWS Secret Access Key

Once this is completed you can continue by installing your SSL certificate in AWS using AWS CLI and AWS IAM.

Installing SSL certificates in AWS IAM

Once you have AWS CLI working with your credentials it’s time to upload your SSL certificate together with the private key and optionally the certificate chain to AWS IAM:

Upload SSL certificate to AWS IAM

aws iam upload-server-certificate \
--server-certificate-name your-certificate-name \
--certificate-body file://server-certificate.pem \
--private-key file://server-key.pem

AWS CLI should respond with an appropriate success message, like so:

{
    "ServerCertificateMetadata": {
        "ServerCertificateId": "SOMEHASHVALUE", 
        "ServerCertificateName": "your-certificate-name", 
        "Expiration": "2018-09-14T23:59:59Z", 
        "Path": "/", 
        "Arn": "arn:aws:iam::SOME_NUMBER:server-certificate/your-certificate-name", 
        "UploadDate": "2017-09-15T09:40:52.183Z"
    }
}

You will now be able to select your SSL certificate when creating your EC2 Load Balancer.

Setup classic AWS EC2 Load Balancer for SSL termination

Now that everything is prepared we are able to actually create the classic EC2 Load Balancer for SSL termination.

Open up the AWS Management Console and create a classic EC2 Load Balancer by selecting the Previous Generation Class Load Balancer:

Then you will be guided to the setup steps for your EC2 classic Load Balancer:

Step 1: Basic Configuration

Step 2: Security Groups

Step 3: Certificate and Cipher Settings

Step 4: Health Check

Step 5: Adding EC2 instances

Step 6: Add Tags

Step 7: Review and Start EC2 Load Balancer

If everything went OK you should be able to access the newly created Load Balancer using https://www.your-server.com now.

Check your SSL setup

Make sure to check your SSL setup using external services such as SSLLabs:

https://www.ssllabs.com/ssltest/analyze.html?d=www.your-server.com

Fixing Magento 1 newsletter queue bug due to missing encoding in grid renderer class

In a recent Magento 1.9.3.2 project we experienced a strange behavior related to the built-in newsletter module in admin grid. When trying to add a newsletter template to the queue using the action dropdown in the admin grid the following JavaScript error showed up:

Uncaught SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at String.parseJSON [as evalJSON] (prototype.js:720)
    at Object.execute (grid.js:717)
    at HTMLSelectElement.onchange (085f35f…:722)

A quick look at the option value for the admin grid’s row select input showed that the JSON was not properly escaped:

<select class="action-select" onchange="varienGridAction.execute(this);"><option value=""></option><option value="{" href":"https:\="" \="" www.someshop.com\="" index.php\="" __ma2ge_a5dm2in__\="" newsletter_queue\="" edit\="" template_id\="" 1\="" key\="" e5bdca9b9185cd175c6f9d297127d238\="" "}"="">Newsletter Warteschlange ...</option><option value="{" popup":true,"href":"https:\="" \="" www.someshop.com\="" index.php\="" __ma2ge_a5dm2in__\="" newsletter_template\="" preview\="" id\="" 1\="" key\="" b4b16e0fa2fb208b6191e6ddb3a6282c\="" ","onclick":"popwin(this.href,'_blank','width="800,height=700,resizable=1,scrollbars=1');return" false;"}"="">Vorschau</option></select>

As you can see the double quotes for the JSON option value was broken, thus resulting in a JavaScript exception when varienGridAction.execute(this) is triggered, e.g.:

<option value="{" href":"https:\="" \=""...

Since the built-in newsletter uses a custom row renderer for this action dropdown a check in Mage_Adminhtml_Block_Widget_Grid_Column_Renderer_Action was required, in particular _toOptionHtml:

  protected function _toOptionHtml($action, Varien_Object $row) {
        $actionAttributes = new Varien_Object();

        $actionCaption = '';
        $this->_transformActionData($action, $actionCaption, $row);
        $htmlAttibutes = array('value'=> $this->escapeHtml(Mage::helper('core')->jsonEncode($action)));

        $actionAttributes->setData($htmlAttibutes);
        return '<option ' . $actionAttributes->serialize() . '>' . $actionCaption . '</option>';
  }

The solution in this case is rather simple:

$htmlAttibutes = array('value'=> htmlentities($this->escapeHtml(Mage::helper('core')->jsonEncode($action))));

Further investigation is needed in this case as to why additional encoding is required. In the meantime the offending class was overwritten with the corresponding local version.

Using Command Query Responsibility Segregation to speed up data-driven web applications

Typical web applications spend most of their time reading data from a data storage (i.e. a database) which is then processed and converted to HTML for the desired frontend visualization.

Furthermore, when relying on programming languages such as PHP that are essentially based on the “shared nothing principle” the performance overhead for consecutive processing of possible the same request flow becomes obvious. Even when deploying modern PHP frameworks the basic flow of events often times remains pretty much the same and rather simple at the core:

  1. determine a controller/action for handling an URL specified in a request
  2. read required data from database based on the data model at hand
  3. process and (pre-)render the view for the frontend, i.e. produce HTML/JavaScript/etc.
  4. send response to client and wait for more to come

Since performance is a key factor for visitors to not bounce we need to make sure to minimize any potential overhead in the process of delivering responses to them.

Horizontal Scaling and Caching

In contrast to PHP, Java or Node.js for instance don’t follow the shared nothing principle and keep read already data in memory that can potentially be shared across requests. So why do we not just use an Application Server such as Apache Tomcat and save all database queries in memory and programmatically decide when to persist data? Wouldn’t this just solve our waiting time for reading data from a database before it get converted and sent back to our clients? Well, this heavily depends on the software and system architecture at hand.

Imagine the simplest case with one application/database server. With the increasing amount of website traffic your server will have to be upgraded (vertical scaling) to meet your increasing memory requirements in order to keep data read accessible without querying your database again.

Horizontal scaling to the rescue?

At some point you will realize that you are forced to do horizontal scaling and add more machines to cope with this situation (which actually is part of designing the software and system architecture). Unfortunately, horizontal scaling adds an additional layer of complexity since you are now required to synchronize your application and data across multiple nodes. Furthermore, you need to make sure which processes are allowed to read and which are allowed to write to your database in order to prevent data-inconsistencies and possible race-conditions.

Caching to the rescue?

In order to improve the performance of web applications we oftentimes deploy additional caching mechanism to reduce database queries and unnecessary frontend rendering steps. Thus, technologies such as Varnish Cache are deployed to serve as a Reverse Proxy and full-page cache (FPC). These setups are required to handle user-specific frontend data too. For instance, imagine an online shop. Once a user logs in (in fact also prior to this) personalized data will be rendered and displayed in the frontend. The FPC will need to be able to cope with this situation too, which Varnish in fact is capable of using Edge Side Includes (ESI).

But, these personalized, dynamic frontend fragments that are rather costly to generate are in fact not part of the actual cache. Furthermore, deploying caching technologies adds an additional frontend layer that needs to be dealt with. Also, in practice purging only parts of a web-application is oftentimes not that easy to achieve. Thus, caching is not the definitive answer for our performance problem.

Command Query Responsibility Segregation to the rescue

Going back one step to our initial problem at hand we need to realize that the part of reading data from the underlying database is costly since it means that we also need to render the frontend response. So the actual goal is to determine when we are in fact required to (re-)generate frontend responses by going through all the steps of our process workflow.

Let’s look at this situation by using an online shop as example. In general, product data is not likely to change with every request. In fact, we only want to (re-)generate frontend data if product data has changed. Thus, we need to differentiate between read and write operations in our data models in order to be able to decide when to trigger a potential (re-)generation of frontend data. This is when Command Query Responsibility Segregation (CQRS) comes into play.

CQRS is an architecture pattern that strictly separates read and write operations. For data models this basically means that we now have two classes instead of one:

  1. class for read operations (“R”)
  2. class for write operations (“W”)

Based on this separation of concerns we are now able to have another closer look at our performance issue at hand, since now only write operations are legible to trigger (re-)generations of frontend data whereas read operations merely serve already existing data.

Frontend Snippets

Going back to our online shop example we now only trigger the (re-)generation of a product page for the frontend once product data is changed (for instance in the ERP). Furthermore, we are able to generate frontend snippets based on this approach too, for instance to generate product item previews on category pages. These snippets need to be generated in a view that they can be persisted to a key-value-store. This way, we are able to quickly load pre-generated frontend snippets from high-performance key-value stores.

Key-Value-Store as simple cache?

You might think that the key-value-store just described is nothing more than a simple cache, right? Wrong! There are fundamental differences between them. From the frontend perspective the key-value-store became the primary data source instead of the underlying database. Thus, if the key-value-store is missing an entry the application will behave as if this entry is missing in the database. Hence, entries in the key-value-store do not have a TTL since they are per definition always up-to-date. The frontend does not know that these entries are updated by external components (through write operations as previously discussed) and it does not care too. Furthermore, the process of superseding entries with newer ones in regular caching solutions such as Varnish also does not happen for the key-value-store.

Micro services for frontend

In order for such an architecture to work efficiently micro services are deployed in the frontend to separate write requests, such as adding items to a cart in online shops. The aforementioned frontend snippets are generated in the background and added to the key-value-store waiting to be updated by write operations. This way, the front- and backend are completely separated and can be scaled independently. So in case you hit a traffic peak you are able to add additional frontend instances in order to tackle potential performance issues beforehand, pretty neat right?

Avoid Duplicate Content by enforcing trailing slash in URLs

Most of today’s popular Content Management Systems (CMS) support the option to use Search Engine Friendly URLs (SEF URLs). This option is either provided by using permalink structures or simply by deploying dynamic URL rewrites based on pre-defined URL schemas (which the aforementioned permalink structures basically are anyway).

Let’s take WordPress or TYPO3 for instance. Both systems ship with a SEF URL feature that can be easily customized to your needs. In WordPress you can set your required URL schemas based on the Permalink Settings as shown below:

WordPress Permalinks
WordPress Permalinks Settings

In TYPO3 you will want to use the popular realurl extension to setup your URL structures and various i18n settings.

Mind Duplicate Content based on multiple URLs

So you’ve setup your CMS to use pretty SEF URLs instead of parameterized ones. Nice! A common mistake when using these URL rewrites mechanisms is the fact that these URLs are by default accessible through at least 3 different URL paths:

  1. URL reference by ID
  2. SEF-URL without trailing slash
  3. SEF-URL with trailing slash

In order to uniquely identify pages, post or resources in general CMS deploy unique identifiers. Thus, by default your resources will be accessible by using the respective unique identifier, e.g.

http://www.yourdomain.com/?p=123

Next, when enabling and configuring SEF-URLs you also need to keep in mind that there are always two variants, the URL with and without a trailing slash:

http://www.yourdomain.com/some-page vs. http://www.yourdomain.com/some-page/

As you can imagine having a resource accessible through multiple URLs makes your site vulnerable to the issue of Duplicate Content (DC). Thus, you should make sure that your URLs are only accessible through a single URL based on a schema of your choice (with/without trailing slash, etc.).

Enforce Trailing slash URLs using .htaccess

My recommendation is to always use SEF URLs with a trailing slash and block access to all other variants to avoid duplicate content and keep your URL space clean. Below you find a snippet to enforce trailing slash URLs using .htaccess for Apache. Of course you can achieve the same behavior for any other web server too, like nginx.

Basically, what we are doing here is to do some preliminary checks on the current request and finally redirect the request to the trailing slash version when needed.

First, let’s only check GET requests here:

RewriteCond %{REQUEST_METHOD} ^GET$

Second, ignore rewrites for existing files:

RewriteCond %{REQUEST_FILENAME} !-f

Third, in case we want to exclude certain paths from rewriting:

RewriteCond %{REQUEST_URI} !^/exclude-me.*$

Fourth, check if we actually need to do a rewrite (hint: you might want to check here for your root page too):

RewriteCond %{REQUEST_URI} !^(.+)/$

Finally, do the rewrite by redirecting to the trailing slash version using a HTTP 301 redirect:

RewriteRule ^(.*[^/])$ /$1/ [L,R=301]

Since there is no all-in-one solution you might need to customize the snippet to your needs but I believe that you get the gist.

Final remarks: Make sure that your generated sitemaps also only use the actual URL variant that you’ve decided to use, for instance the trailing slash version. Otherwise the various search engine crawlers will not be amused to be redirected on every entry in your sitemap. Also, make sure that you don’t use !^(.*)/$ to check for existing trailing slash URLs as this expression also matches the root directory (* instead of + in the expression to only match one or more characters in front of the trailing slash).

Enjoy!

Software project types are visible to all JIRA users

By default, Software project types in JIRA are visible to all JIRA users. This might pose a potential security problem since all your JIRA users will be able to see and possibly access your company’s software projects. Having setup and customized JIRA for numerous customers and projects in the past this is one of the first issues you should definitely deal with from a security perspective. This post serves as quick note on how to disable access to all JIRA users for software projects types by default and configure access on project based settings instead.

Disable access to all JIRA users for software project types

Per default, the permission to browse Software project types in JIRA defaults to all users. to To overcome this potential security issue go ahead and have a look at the Browse Projects permission in your Default software Schema via the Permission Scheme settings, as shown below:

JIRA Default software scheme
JIRA Default software scheme

As you can see by default the Browse Projects permission includes Application access for Any logged in user. Since this setting supersedes the Project role related setting all of your JIRA users will be able to see your software projects by either browsing through the projects list (hence the permission named “Browse Projects”) or by simply using direct links.

Now since we want to set permissions on project level for our users we need to remove the Browse Projects permission setting for the Application access and only use the Project role instead. The screenshot below shows the correct setting for project based browse permissions for your JIRA projects:

JIRA Software Project Scheme - edited
JIRA Default software scheme – edited

You are now able to properly configure access to your JIRA software projects on a project based level and your JIRA users will only see those projects they are a member of.

Final hint: Make sure to check your Browse Projects permission for all of your remaining permission schemes. As always, enjoy JIRA 🙂

Magento acquires RJMetrics to add Magento Analytics to portfolio

Magento recently acquired RJMetrics, a powerful cloud-based analytics solution for e-commerce merchants. RJMetrics will be added to the Magento product suite as a brand new solution called Magento Analytics.

RJMetrics

The idea behind Magento Analytics according to Magento is that it will allow non-technical business users to quickly and easily integrate with enterprise-grade data sets across a broad array of applications in order to consolidate and analyze data for effective multi-brand, cross-channel reporting. Hereby, RJMetrics will be used to directly integrate an analytics solution to the core Magento framework to enable deep customer analysis for improve possible customization, personalization and merchandising, or as Magento puts it in fancy terms to continuously optimize digital shopping experiences and performance to increase sales and gross margin.

RJMetrics, or better put Magento Analytics will need to be put to a test to compare it to existing analytics and business intelligence solutions. After all, existing analytics tools such as Google Analytics or the open source alternative Piwik that although primarily focused on web tracking and web analytics with the correct setup can be used to effectively gather customer insights too without the need to incorporate additional tools in existing deployments.

RJMetrics key features

RJMetrics key features are Cohort Analysis to measure the effects of defined growth efforts on customer behavior, from conversion rate optimization to customer loyalty programs, Churn Analysis in order to adapt to customer feedback and identify causes that are making your customers churn, as always Marketing ROI to identify those channels and campaigns that are valuable for your business and be able to calculate ROI based on customer acquisition cost and customer lifetime value, Revenue Analytics based on your raw shop data, Email Segmentation to effectively target customers based on their behavior and finally Holiday Performance Enablers to optimize ad spendings on your key shopping days, ensure stocks are at hand and finally plan potential follow-up strategies to turn holiday buyers into loyal customers.

It’s good to know that Magento is expanding its horizon by acquiring RJMetrics and presenting a viable in-house analytics solution. For those of you who have tested or even implemented RJMetrics in your setups I’d be glad to hear some feedback. For more information about RJMetrics and what Magento Analytics can offer have a look at the RJMetrics website.

Enable command and branch name autocompletion for GIT

When working in GIT based software projects that have a broad range of GIT branches the need for autocompletion of these branch names and GIT commands in general in the shell is one of the first tasks you want to setup for your developers (and yourself ;). After all it’s all about removing impediments to get highly motivated developers who produce high quality code in return, isn’t it?

Setup git-completion.bash in your shell to enable command and branch name autocompletion

By default GIT ships a file called git-completion.bash (at least on Linux, Mac and most probably other major platforms as well) that when setup in your shell will autocomplete commands and branch names, nice right! This file actually represent a comprehensive ruleset that will autocomplete all kinds of GIT commands, try it out you will be amazed.

Normally you find this file in contrib/complete/git-completion.bash but you can also download it from github.

All you have to do to enable branch name and command autocompletion for GIT is to add a source reference to git-completion.bash in your shell’s profile (~/.bashrc or ~/.profile). So open up your shell’s profile (I’ve chosen ~/.profile here)

vi ~/.profile

and add a source call to the git-completion.bash file

source ~/.git-completion

In case there’s no immediate effect do a manual source call on your shell profile file:

$ source ~/.profile

This should be all to enable branch name autocompletion for GIT. Enjoy your developers’ and your new productivity!

Magento Newsletter Unsubscribe Form

One of the most missed functions in Magento 1 is a newsletter unsubscribe form. Although, by default you can generate newsletter unsubscribe links to be used for instance in newsletters sent out to subscribers there’s no nice out-of-the-box way to integrate a newsletter unsubscribe form in Magento 1.

Which is why there already exist some extensions on the Magento Connect Marketplace that extend the existing newsletter functionality by e.g. adding a simple unsubscribe field and toggling the action submitted (“subscribe” vs. “unsubscribe”). Unfortunately, having a look at most of the existing extensions made clear that they either include too much functionality or simply are outdated.

Implementing the Magento Newsletter Unsubscribe Extension

Luckily, implementing the newsletter unsubscribe functionality in Magento is pretty straight forward. In order to share this information here’s the gist on how to create a Magento Newsletter unsubscribe form extension.

Basically, what we are trying to achieve here is an additional Magento page with a distinct URL that incorporates our newsletter unsubscribe form, i.e. “newsletter-unsubscribe”. This form handles submissions and checks if the e-mail submitted currently is subscribed to the Magento newsletter. If that’s the case it will unsubscribe this e-mail based on the built-in Magento functionality, i.e. it will automatically sent out the un-subscription confirmation mail and execute any other events that you might have integrated for newsletter related events. Last but not least, as always everything should be translatable.

Custom route instead of CMS page

Instead of creating an additional CMS page and loading our newsletter unsubscribe form into it (e.g. using layout updates) we are going to create a custom route for our extension. By default, this route will be newsletter-unsubscribe. The relevant entries in the config.xml are as follows:

<?xml version="1.0"?>
<config>
 
  <modules>
    <BothInteract_NewsletterUnsubscribe>
      <version>1.0.1</version>
    </BothInteract_NewsletterUnsubscribe>
  </modules> 
 
  <frontend>
    <routers>
      <bothinteract_newsletterunsubscribe>
        <use>standard</use>
        <args>
          <module>BothInteract_NewsletterUnsubscribe</module>
          <frontName>newsletter-unsubscribe</frontName>
        </args>
      </bothinteract_newsletterunsubscribe>
    </routers>
 
    <secure_url>
      <bothinteract_newsletterunsubscribe>/newsletter-unsubscribe/</bothinteract_newsletterunsubscribe>
    </secure_url>

    ...
</config>

As you can see an additional frontend route is added (newsletter-unsubscribe) that maps to our extensions’s controller.

Display newsletter unsubscribe form

Now that URLs starting with “newsletter-unsubscribe” are routed to our controller we need to make sure that our custom block is loaded and unsubscribe submissions are processed correctly.

As always, the default action is index and in our case it’s responsible for rendering or newsletter unsubscribe form:

public function indexAction() {

  $this->loadLayout();

  // load 1-column page layout
  $this->getLayout()
    ->getBlock('root')
    ->setTemplate('page/1column.phtml');

  // load breadcrumb bar
  $breadcrumbs = $this->getLayout()->getBlock('breadcrumbs');
  $helper = Mage::app()->getHelper('bothinteract_newsletterunsubscribe');

  if ($breadcrumbs) {
    $breadcrumbs->addCrumb('home', array(
      'label' => $helper->__('Home'),
      'title' => $helper->__('Go to Home Page'),
      'link' => Mage::getBaseUrl()));
    $breadcrumbs->addCrumb('newsletter-unsubscribe', array(
      'label' => $helper->__('Newsletter Unsubscribe'),
      'title' => $helper->__('Newsletter Unsubscribe'),
      'link' => Mage::getUrl('newsletter-unsubscribe')));
  }

  // render our custom block
  $block = $this->getLayout()
    ->createBlock('bothinteract_newsletterunsubscribe/customblock')
    ->setTemplate('bothinteract_newsletterunsubscribe/form.phtml');

  // append to CMS content block
  $this->getLayout()->getBlock('content')->append($block);

  $this->renderLayout();
}

There’s no magic here but let’s quickly dissect what is happening:

First, we are loading the layout and setting the 1-column page layout by default .

Hint: Using additional extension backend options we can make these settings available for users to edit directly without touching the code. This will be available in version 1.1.

Since we want our page to appear in the breadcrumbs too we manually add it. Finally, we create and append our custom block while setting the corresponding template and render the final layout.

Process newsletter unsubscribe form submissions

Now that we are able to display the newsletter unsubscribe form it’s time to process form submissions. This is done using the unsubscribe action:

public function unsubscribeAction() {
  if ($this->getRequest()->isPost() && $this->getRequest()->getPost('email')) {
    $session = Mage::getSingleton('core/session');
    $email = (string) $this->getRequest()->getPost('email');

  try {
    if (!Zend_Validate::is($email, 'EmailAddress')) {
      Mage::throwException($this->__('Please enter a valid email address.'));
    }

    $subscriber = Mage::getModel('newsletter/subscriber')
      ->loadByEmail($email);

    if ($subscriber && $subscriber->getId()) {
      // check already unsubscribed/inactivated before
      if ($subscriber->getStatus() == Mage_Newsletter_Model_Subscriber::STATUS_NOT_ACTIVE ||
        $subscriber->getStatus() == Mage_Newsletter_Model_Subscriber::STATUS_UNSUBSCRIBED) {
        $session->addSuccess($this->__('You are not registered with this e-mail.'));
      } else {
        $subscriber->unsubscribe();
        $session->addSuccess($this->__('You have been successfully unsubscribed.'));
      }
    } else {
      $session->addSuccess($this->__('You are not registered with this e-mail.'));
    }
   } catch (Mage_Core_Exception $e) {
     $session->addException($e, $this->__('There was a problem with the unsubscription: %s', $e->getMessage()));
   } catch (Exception $e) {
     $session->addException($e, $this->__('There was a problem with the unsubscription.'));
   }
  }
  $this->_redirectReferer();
}

Again, no magic is happening here. We are simply checking the submitted e-mail, try to load the corresponding newsletter subscriber object and check it’s status. Depending on the status we are able to unsubscribe this e-mail or display the appropriate message.

Our actual custom Magento block is very slim on purpose and only returns the form action URL of our custom extension:

class BothInteract_NewsletterUnsubscribe_Block_Customblock extends Mage_Core_Block_Template {

  public function getFormActionUrl() {
    return $this->getUrl('newsletter-unsubscribe/index/unsubscribe/', array('_secure' => true));
  }
}

Yes, that’s all there is to it. Sorry to disappoint you in case you are expecting magic stuff happening here. Thus, based on Magento’s stable architecture adding a newsletter unsubscribe form can be easily achieved.

Download Magento Newsletter Unsubscribe Extension

This extension has been released by Both Interact and is available for free on the Magento Connect marketplace as Both Interact Newsletter Unsubscribe extension. Feel free to download and rate the extension.

 

Fix requested URI error when using owncloud and PHP 7

This is just a quick note for people experiencing problems using ownCloud with PHP 7.0.6 (and potentially versions above) which cause the following error when trying to access the admin backend:

{“reqId”:”qKglxvKVntjfjDJur2Zw”,”remoteAddr”:””,”app”:”index”,”message”:”Exception: {\”Exception\”:\”Exception\”,\”Message\”:\”The requested uri() cannot be processed by the script ‘\\\/apps\\\/owncloud\\\/index.php’)\”,\”Code\”:0,\”Trace\”:\”#0 \\\/apps\\\/owncloud\\\/lib\\\/private\\\/appframework\\\/http\\\/request.php(640): OC\\\\AppFramework\\\\Http\\\\Request->getRawPathInfo()\\n#1 \\\/apps\\\/owncloud\\\/lib\\\/base.php(819): OC\\\\AppFramework\\\\Http\\\\Request->getPathInfo()\\n#2 \\\/apps\\\/owncloud\\\/index.php(39): OC::handleRequest()\\n#3 {main}\”,\”File\”:\”\\\/apps\\\/owncloud\\\/lib\\\/private\\\/appframework\\\/http\\\/request.php\”,\”Line\”:614}”,”level”:3,”time”:”2016-05-07T07:18:43+00:00″}

There’s a simple solution to this problem by patching __isset() in  lib/private/AppFramework/Http/Request.php:

public function __isset($name) {
 + if (in_array($name, $this->allowedKeys, true)) {
 + return true;
 + }
 return isset($this->items['parameters'][$name]);
 }

For more info have a look at the offical GitHub patch commit.

 

PHP Mailing List – Version 3.2.0 released

Today, version 3.2.0 of PHP Mailing List was released.

Version 3.2.0 of PHP Mailing List is a feature release that was focused on further improving existing anti-spam measures and the usability in the admin interface concerning re-sending membership invitations to pending members.

Short URLs using Google URL shortener API

Since a couple of users reported problems when sending mails to Microsoft based mail services, such as outlook.com and hotmail.com, PHP Mailing List now offers the possibility to automatically shorten action URLs using the Google URL shortener API.

Thus, when activating the URL shortener feature PHP Mailing List will automatically shorten reply-to and authorization links. This way URLs generated by PHP Mailing List will not look as “spammy” as they might do now, especially when using the Google event tracking feature introduced in version 2.0.0.

So, in case you suffer from Microsoft related bounce messages such as

<xyz@hotmail.com>:
Connected to 65.54.188.110 but sender was rejected.
Remote host said: 550 SC-001 (BAY004-MC3F36) Unfortunately, messages from x.x.x.x weren't sent. Please contact your Internet service provider since part of their network is on our block list. You can also refer your provider to http://mail.live.com/mail/troubleshooting.aspx#errors.

make sure to activate to URL shortener feature in PHP Mailing List and re-check your sender score and check for possible listings in various blacklists.

Re-send membership invitation to pending members

In addition, due to numerous requests it is now also possible to re-send membership invitations to already pending members. Simply open up the admin interface and hit the re-send link in the pending invitations screen.

Download PHP Mailing List 3.2.0

As always, feel free to download the latest version of PHP Mailing List via GitHub. Comments and questions are welcome.

Sample SEO Magento robots.txt file

Since I get a lot of requests for a robots.txt file designed for Magento SEO here is a sample to get you started. This Magento robots.txt makes the following assumptions:

  • We don’t differentiate between search engines, hence User-agent: *
  • We allow assets to be crawled
    • i.e. images, CSS and JavaScript files
  • We only allow SEF URLs set in Magento
    • e.g. no direct access to the front controller index.php, view categories and products by ID, etc.
  • We don’t allow filter URLs
    • Please note: The list provided is not complete. In case you have custom extension that use filtering make sure to include these filter URLs and parameters in the filter URLs section.
  • We don’t allow session related URL segments
    • e.g. product comparison, customer, etc.
  • We don’t allow specific files to be crawled
    • e.g. READMEs, cron related files, etc.

Magento robots.txt

Enough of the talking, here comes your SEO Magento robots.txt:

# Crawlers Setup
User-agent: *

# Directories
Disallow: /app/
Disallow: /cgi-bin/
Disallow: /downloader/
Disallow: /includes/
Disallow: /lib/
Disallow: /pkginfo/
Disallow: /report/
Disallow: /shell/
Disallow: /var/

# Paths (clean URLs)
Disallow: /index.php/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /catalogsearch/
#Disallow: /checkout/
Disallow: /control/
Disallow: /contacts/
Disallow: /customer/
Disallow: /customize/
Disallow: /newsletter/
Disallow: /poll/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/
Disallow: /catalog/product/gallery/

# Misc. files you don’t want search engines to crawl
Disallow: /cron.php
Disallow: /cron.sh
Disallow: /composer.json
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /STATUS.txt
Disallow: /mage
#Disallow: /modman
#Disallow: /n98-magerun.phar
Disallow: /scheduler_cron.sh
Disallow: /*.php$

# Disallow filter urls
Disallow: /*?min*
Disallow: /*?max*
Disallow: /*?q*
Disallow: /*?cat*
Disallow: /*?manufacturer_list*
Disallow: /*?tx_indexedsearch

Feel free to leave comments below for additional remarks and suggestions for improvement.

Updating Magento tax rules causes code already exists error

When trying to update tax rules in Magento that have been imported by extensions such as firegento magesetup you might run into problems related to duplicate tax rule codes, e.g. “20% VAT” for both customers and retailers.

By default, Magento happily accepts duplicate tax rule codes when importing them trough setup scripts. But when trying to update these tax rules through adminhtml it will fail due to model checks with a “code already exists” error, as shown below:

Magento tax rule code already exists

The easiest solution here is to manually set the tax rule codes in your table tax_calculation_rule. Simply edit the code column there and you are all set. Happy tax’ing!

New table column not updating in Magento

So you’ve added a custom column for a table, e.g. admin_user or through your custom extension’s model table. The only problem is your values are not being stored by your underlying model classes, i.e. abstractions of Mage_Core_Model_Abstract. But why you ask since Magento is supposed to automatically infer database table to model relations through Zend_Db and magic getters and setters (simply put).

Clear that cache

Well, in fact the solution is pretty simple:

Zend_Db caches the structure of the database tables.

Thus, if you add new columns make sure to refresh the cache. Zend will then update the cache accordingly.

Also note that even if you all your caches are disabled the database structure and other info may still be cached in var/cache, so:

Clear /var/cache.

to make sure that your table adaptions reflect properly.