Posted on 1 Comment

Setting up Collabora CODE with NextCloud using Apache reverse proxy on Debian 8 Jessie

Setting up Collabora Online Development Edition (CODE) can be a little tricky. This guide shows the steps needed to get Collabora CODE working using an Apache 2.2 reverse proxy on Debian 8 Jessie.

The steps we are going to have a look at are as follows:

  1. Setup Apache reverse proxy
  2. Setup Collabora CODE based on official Docker image
  3. Install and configure NextCloud Collabora CODE plugin

The basic configuration we are trying to achieve here is:

  1. Setup a secure domain for accessing Collabora from NextCloud
    1. This will be https://office.yourserver.com
    2. I’m assuming that you already have a working SSL certificate for this domain. If not, have a look at Let’s Encrypt in case you want a cheap solution. For Debian 8 Jessie have a look at the certbot guide.
  2. Setup a reverse proxy configuration for this domain that fowards requests to Collabora webservice endpoints
  3. Access Collabora CODE Docker container through Apache reverse proxy which itself exclusively listens an a secure line on Port 9980 (default)
  4. Install the Collabora NextCloud plugin and configure it to access Collabora through our reverse proxy

Setup Collabora CODE based on official Docker image

Collabora can be either installed using a package provided by your distribution or by using the official Docker image collabora/code.

Normally, the Docker container setup should be pretty pain free. Having said that, for Debian 8 Jessie you need to adjust the storage driver to devicemapper as it seems that the default docker storage driver AUFS and Debian 8 do not work together.

Adjust Docker storage driver to devicemapper

The steps required are again pretty straight forward. First get current ExecStart from your docker.service file:

grep ExecStart /lib/systemd/system/docker.service

Example output:
ExecStart=/usr/bin/dockerd -H fd://

Then use this result to create a systemd Docker drop-in configuration file and create the service directory first if it does not yet exist as well:

mkdir /etc/systemd/system/docker.service.d
editor /etc/systemd/system/docker.service.d/execWithDeviceMapper.conf

Put the following content in execWithDeviceMapper.conf:

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd --storage-driver=devicemapper -H fd://.

Finally, restart systemd, docker.service and possibly your existing Collabora container if you had one running:

systemctl daemon-reload
systemctl restart docker.service

Disclaimer: For higher volume production sites you definitely want to optimize this setup (…).

The command the start your Collabora Docker container is as follows:

docker run -t -d -p 127.0.0.1:9980:9980 \
-e 'domain=www\\.yournextcloud1\\.com\|www\\.yournextcloud2\\.com' \
--restart always --cap-add MKNOD collabora/code

Note that I’ve provided two domains in the above command to show how to enable multiple domains to access your Collabora web service.

As always, since the Docker container starts in detached mode make sure to check for possible problems using

docker logs YOUR_CONTAINER_ID

Now that we have Collabora CODE up and running as Docker container we need to make it available to the outside world using an Apache reverse proxy.

Setup Apache reverse proxy

First and foremost, I will not cover the exact steps to setup the base Apache web server here but provide a working vhost configuration.

Required Apache modules

The additional Apache module requirements to get Apache working as reverse proxy for Collabora CODE are:

  1. mod_proxy

  2. mod_proxy_http

  3. mod_proxy_wstunnel

  4. mod_ssl

Apart from mod_proxy_wstunnel the configuration steps should be pretty straight forward. When using Apache 2.2 and mod_proxy_wstunnel on the other hand things can get a little more tricky since you need to apply a patch and compile the module yourself. Have a look at the very handy guide by waleedsamy on github to compile mod_proxy_wstunnel yourself.

Apache Reverse Proxy vhost configuration

Once all requirements are satisfied we can setup the vhost configuration for the Apache reverse proxy domain.

Remember, our internet-facing domain for accessing Collabora CODE will be office.yourserver.com. This will be the basis for your vhost configuration below.

UseCanonicalName off
ServerName office.yourserver.com

# Enable and configure SSL/TLS
SSLEngine on
SSLCertificateFile yourserver-cert
SSLCertificateKeyFile yourserver-key
SSLCertificateChainFile yourserver-cacert

SSLProtocol all -SSLv2 -SSLv3
SSLCipherSuite ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES25$
SSLHonorCipherOrder on
SetEnvIf User-Agent ".*MSIE.*" \
nokeepalive ssl-unclean-shutdown \
downgrade-1.0 force-response-1.0

# Encoded slashes need to be allowed
AllowEncodedSlashes NoDecode

# Enable and configure SSL Proxy
SSLProxyEngine On
SSLProxyVerify None
SSLProxyCheckPeerCN Off

# Make sure to keep the host
ProxyPreserveHost On

# static html, js, images, etc. served from loolwsd
# loleaflet is the client part of LibreOffice Online
ProxyPass /loleaflet https://127.0.0.1:9980/loleaflet retry=0
ProxyPassReverse /loleaflet https://127.0.0.1:9980/loleaflet

# WOPI discovery URL
ProxyPass /hosting/discovery https://127.0.0.1:9980/hosting/discovery retry=0
ProxyPassReverse /hosting/discovery https://127.0.0.1:9980/hosting/discovery

# Main websocket
ProxyPassMatch "/lool/(.*)/ws$" wss://127.0.0.1:9980/lool/$1/ws nocanon

# Admin Console websocket
ProxyPass /lool/adminws wss://127.0.0.1:9980/lool/adminws

# Download as, Fullscreen presentation and Image upload operations
ProxyPass /lool https://127.0.0.1:9980/lool
ProxyPassReverse /lool https://127.0.0.1:9980/lool

Check if your reverse proxy is working by accessing the WOPI discovery URL:

https://office.yourserver.com/hosting/discovery

If that gives you the corresponding XML namespace information you should be good to go.

Install and configure NextCloud Collabora CODE plugin

This is the last step required and should be the easiest one.

  1. Go to the Apps section and choose “Office & Text”
  2. Install the “Collabora Online” app
  3. In Admin -> Collabora Online specific the server you have setup before (https://office.yourserver.com)

Finally, try to create and edit a document via NextCloud. Enjoy your private Collabora setup using NextCloud!

For more information have a look at the official Collabora CODE documentation.

Posted on Leave a comment

Setting up a classic AWS EC2 Load Balancer for SSL termination

Amazon AWS Logo

Setting up an AWS EC2 Load Balancer for SSL termination is pretty straight forward once you have all required information at hand.

In this example we will be setting up a classic AWS EC2 Load Balancer for SSL termination.

Before we start configuring the load balancer make sure you have the following required assets at hand:

  1. Server SSL private key
  2. Server SSL certificate
  3. Server SSL certificate chain (if required by your SSL setup)
  4. AWS CLI working
    1. Access to IAM via CLI
    2. AWS Access Key ID
    3. AWS Secret Access Key
  5. Access to the AWS Management Console to setup classic EC2 Load Balancer
  6. Optional: a cup of coffee or tea 😉

We will now go through all the steps required to get your EC2 Load Balancer for SSL termination up and running in no time.

Converting SSL certificates to PEM format

First and foremost, your Server SSL private key and your Server SSL certificate are required to be in PEM format.

If this is not the case use the following commands to convert your private key and certificate to PEM format:

Convert SSL private key to PEM format

openssl rsa -in server-key.key -text > server-key.pem

Convert SSL certificate to PEM format

openssl x509 -inform PEM -in server-certificate.crt > server-certificate.pem

Setup AWS CLI

In order to set up AWS CLI you may use the bundle provided by Amazon or install the requirements manually.

After the installation you will need to setup your AWS CLI credentials.

Make sure that the Policy IAMFullAccess is set for the target AWS CLI user:

Then, setup AWS CLI credentials using the following command:

aws configure

You will be prompted to enter your

  1. AWS Access Key ID

  2. AWS Secret Access Key

Once this is completed you can continue by installing your SSL certificate in AWS using AWS CLI and AWS IAM.

Installing SSL certificates in AWS IAM

Once you have AWS CLI working with your credentials it’s time to upload your SSL certificate together with the private key and optionally the certificate chain to AWS IAM:

Upload SSL certificate to AWS IAM

aws iam upload-server-certificate \
--server-certificate-name your-certificate-name \
--certificate-body file://server-certificate.pem \
--private-key file://server-key.pem

AWS CLI should respond with an appropriate success message, like so:

{
    "ServerCertificateMetadata": {
        "ServerCertificateId": "SOMEHASHVALUE", 
        "ServerCertificateName": "your-certificate-name", 
        "Expiration": "2018-09-14T23:59:59Z", 
        "Path": "/", 
        "Arn": "arn:aws:iam::SOME_NUMBER:server-certificate/your-certificate-name", 
        "UploadDate": "2017-09-15T09:40:52.183Z"
    }
}

You will now be able to select your SSL certificate when creating your EC2 Load Balancer.

Setup classic AWS EC2 Load Balancer for SSL termination

Now that everything is prepared we are able to actually create the classic EC2 Load Balancer for SSL termination.

Open up the AWS Management Console and create a classic EC2 Load Balancer by selecting the Previous Generation Class Load Balancer:

Then you will be guided to the setup steps for your EC2 classic Load Balancer:

Step 1: Basic Configuration

Step 2: Security Groups

Step 3: Certificate and Cipher Settings

Step 4: Health Check

Step 5: Adding EC2 instances

Step 6: Add Tags

Step 7: Review and Start EC2 Load Balancer

If everything went OK you should be able to access the newly created Load Balancer using https://www.your-server.com now.

Check your SSL setup

Make sure to check your SSL setup using external services such as SSLLabs:

https://www.ssllabs.com/ssltest/analyze.html?d=www.your-server.com

Posted on Leave a comment

Fixing Magento 1 newsletter queue bug due to missing encoding in grid renderer class

Magento Logo

In a recent Magento 1.9.3.2 project we experienced a strange behavior related to the built-in newsletter module in admin grid. When trying to add a newsletter template to the queue using the action dropdown in the admin grid the following JavaScript error showed up:

Uncaught SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at String.parseJSON [as evalJSON] (prototype.js:720)
    at Object.execute (grid.js:717)
    at HTMLSelectElement.onchange (085f35f…:722)

A quick look at the option value for the admin grid’s row select input showed that the JSON was not properly escaped:

<select class="action-select" onchange="varienGridAction.execute(this);"><option value=""></option><option value="{" href":"https:\="" \="" www.someshop.com\="" index.php\="" __ma2ge_a5dm2in__\="" newsletter_queue\="" edit\="" template_id\="" 1\="" key\="" e5bdca9b9185cd175c6f9d297127d238\="" "}"="">Newsletter Warteschlange ...</option><option value="{" popup":true,"href":"https:\="" \="" www.someshop.com\="" index.php\="" __ma2ge_a5dm2in__\="" newsletter_template\="" preview\="" id\="" 1\="" key\="" b4b16e0fa2fb208b6191e6ddb3a6282c\="" ","onclick":"popwin(this.href,'_blank','width="800,height=700,resizable=1,scrollbars=1');return" false;"}"="">Vorschau</option></select>

As you can see the double quotes for the JSON option value was broken, thus resulting in a JavaScript exception when varienGridAction.execute(this) is triggered, e.g.:

<option value="{" href":"https:\="" \=""...

Since the built-in newsletter uses a custom row renderer for this action dropdown a check in Mage_Adminhtml_Block_Widget_Grid_Column_Renderer_Action was required, in particular _toOptionHtml:

  protected function _toOptionHtml($action, Varien_Object $row) {
        $actionAttributes = new Varien_Object();

        $actionCaption = '';
        $this->_transformActionData($action, $actionCaption, $row);
        $htmlAttibutes = array('value'=> $this->escapeHtml(Mage::helper('core')->jsonEncode($action)));

        $actionAttributes->setData($htmlAttibutes);
        return '<option ' . $actionAttributes->serialize() . '>' . $actionCaption . '</option>';
  }

The solution in this case is rather simple:

$htmlAttibutes = array('value'=> htmlentities($this->escapeHtml(Mage::helper('core')->jsonEncode($action))));

Further investigation is needed in this case as to why additional encoding is required. In the meantime the offending class was overwritten with the corresponding local version.

Posted on Leave a comment

Using Command Query Responsibility Segregation to speed up data-driven web applications

Performance

Typical web applications spend most of their time reading data from a data storage (i.e. a database) which is then processed and converted to HTML for the desired frontend visualization.

Furthermore, when relying on programming languages such as PHP that are essentially based on the “shared nothing principle” the performance overhead for consecutive processing of possible the same request flow becomes obvious. Even when deploying modern PHP frameworks the basic flow of events often times remains pretty much the same and rather simple at the core:

  1. determine a controller/action for handling an URL specified in a request
  2. read required data from database based on the data model at hand
  3. process and (pre-)render the view for the frontend, i.e. produce HTML/JavaScript/etc.
  4. send response to client and wait for more to come

Since performance is a key factor for visitors to not bounce we need to make sure to minimize any potential overhead in the process of delivering responses to them.

Horizontal Scaling and Caching

In contrast to PHP, Java or Node.js for instance don’t follow the shared nothing principle and keep read already data in memory that can potentially be shared across requests. So why do we not just use an Application Server such as Apache Tomcat and save all database queries in memory and programmatically decide when to persist data? Wouldn’t this just solve our waiting time for reading data from a database before it get converted and sent back to our clients? Well, this heavily depends on the software and system architecture at hand.

Imagine the simplest case with one application/database server. With the increasing amount of website traffic your server will have to be upgraded (vertical scaling) to meet your increasing memory requirements in order to keep data read accessible without querying your database again.

Horizontal scaling to the rescue?

At some point you will realize that you are forced to do horizontal scaling and add more machines to cope with this situation (which actually is part of designing the software and system architecture). Unfortunately, horizontal scaling adds an additional layer of complexity since you are now required to synchronize your application and data across multiple nodes. Furthermore, you need to make sure which processes are allowed to read and which are allowed to write to your database in order to prevent data-inconsistencies and possible race-conditions.

Caching to the rescue?

In order to improve the performance of web applications we oftentimes deploy additional caching mechanism to reduce database queries and unnecessary frontend rendering steps. Thus, technologies such as Varnish Cache are deployed to serve as a Reverse Proxy and full-page cache (FPC). These setups are required to handle user-specific frontend data too. For instance, imagine an online shop. Once a user logs in (in fact also prior to this) personalized data will be rendered and displayed in the frontend. The FPC will need to be able to cope with this situation too, which Varnish in fact is capable of using Edge Side Includes (ESI).

But, these personalized, dynamic frontend fragments that are rather costly to generate are in fact not part of the actual cache. Furthermore, deploying caching technologies adds an additional frontend layer that needs to be dealt with. Also, in practice purging only parts of a web-application is oftentimes not that easy to achieve. Thus, caching is not the definitive answer for our performance problem.

Command Query Responsibility Segregation to the rescue

Going back one step to our initial problem at hand we need to realize that the part of reading data from the underlying database is costly since it means that we also need to render the frontend response. So the actual goal is to determine when we are in fact required to (re-)generate frontend responses by going through all the steps of our process workflow.

Let’s look at this situation by using an online shop as example. In general, product data is not likely to change with every request. In fact, we only want to (re-)generate frontend data if product data has changed. Thus, we need to differentiate between read and write operations in our data models in order to be able to decide when to trigger a potential (re-)generation of frontend data. This is when Command Query Responsibility Segregation (CQRS) comes into play.

CQRS is an architecture pattern that strictly separates read and write operations. For data models this basically means that we now have two classes instead of one:

  1. class for read operations (“R”)
  2. class for write operations (“W”)

Based on this separation of concerns we are now able to have another closer look at our performance issue at hand, since now only write operations are legible to trigger (re-)generations of frontend data whereas read operations merely serve already existing data.

Frontend Snippets

Going back to our online shop example we now only trigger the (re-)generation of a product page for the frontend once product data is changed (for instance in the ERP). Furthermore, we are able to generate frontend snippets based on this approach too, for instance to generate product item previews on category pages. These snippets need to be generated in a view that they can be persisted to a key-value-store. This way, we are able to quickly load pre-generated frontend snippets from high-performance key-value stores.

Key-Value-Store as simple cache?

You might think that the key-value-store just described is nothing more than a simple cache, right? Wrong! There are fundamental differences between them. From the frontend perspective the key-value-store became the primary data source instead of the underlying database. Thus, if the key-value-store is missing an entry the application will behave as if this entry is missing in the database. Hence, entries in the key-value-store do not have a TTL since they are per definition always up-to-date. The frontend does not know that these entries are updated by external components (through write operations as previously discussed) and it does not care too. Furthermore, the process of superseding entries with newer ones in regular caching solutions such as Varnish also does not happen for the key-value-store.

Micro services for frontend

In order for such an architecture to work efficiently micro services are deployed in the frontend to separate write requests, such as adding items to a cart in online shops. The aforementioned frontend snippets are generated in the background and added to the key-value-store waiting to be updated by write operations. This way, the front- and backend are completely separated and can be scaled independently. So in case you hit a traffic peak you are able to add additional frontend instances in order to tackle potential performance issues beforehand, pretty neat right?

Posted on Leave a comment

Software project types are visible to all JIRA users

Jira Logo

By default, Software project types in JIRA are visible to all JIRA users. This might pose a potential security problem since all your JIRA users will be able to see and possibly access your company’s software projects. Having setup and customized JIRA for numerous customers and projects in the past this is one of the first issues you should definitely deal with from a security perspective. This post serves as quick note on how to disable access to all JIRA users for software projects types by default and configure access on project based settings instead.

Disable access to all JIRA users for software project types

Per default, the permission to browse Software project types in JIRA defaults to all users. to To overcome this potential security issue go ahead and have a look at the Browse Projects permission in your Default software Schema via the Permission Scheme settings, as shown below:

JIRA Default software scheme
JIRA Default software scheme

As you can see by default the Browse Projects permission includes Application access for Any logged in user. Since this setting supersedes the Project role related setting all of your JIRA users will be able to see your software projects by either browsing through the projects list (hence the permission named “Browse Projects”) or by simply using direct links.

Now since we want to set permissions on project level for our users we need to remove the Browse Projects permission setting for the Application access and only use the Project role instead. The screenshot below shows the correct setting for project based browse permissions for your JIRA projects:

JIRA Software Project Scheme - edited
JIRA Default software scheme – edited

You are now able to properly configure access to your JIRA software projects on a project based level and your JIRA users will only see those projects they are a member of.

Final hint: Make sure to check your Browse Projects permission for all of your remaining permission schemes. As always, enjoy JIRA 🙂

Posted on Leave a comment

Magento acquires RJMetrics to add Magento Analytics to portfolio

Magento Logo

Magento recently acquired RJMetrics, a powerful cloud-based analytics solution for e-commerce merchants. RJMetrics will be added to the Magento product suite as a brand new solution called Magento Analytics.

RJMetrics

The idea behind Magento Analytics according to Magento is that it will allow non-technical business users to quickly and easily integrate with enterprise-grade data sets across a broad array of applications in order to consolidate and analyze data for effective multi-brand, cross-channel reporting. Hereby, RJMetrics will be used to directly integrate an analytics solution to the core Magento framework to enable deep customer analysis for improve possible customization, personalization and merchandising, or as Magento puts it in fancy terms to continuously optimize digital shopping experiences and performance to increase sales and gross margin.

RJMetrics, or better put Magento Analytics will need to be put to a test to compare it to existing analytics and business intelligence solutions. After all, existing analytics tools such as Google Analytics or the open source alternative Piwik that although primarily focused on web tracking and web analytics with the correct setup can be used to effectively gather customer insights too without the need to incorporate additional tools in existing deployments.

RJMetrics key features

RJMetrics key features are Cohort Analysis to measure the effects of defined growth efforts on customer behavior, from conversion rate optimization to customer loyalty programs, Churn Analysis in order to adapt to customer feedback and identify causes that are making your customers churn, as always Marketing ROI to identify those channels and campaigns that are valuable for your business and be able to calculate ROI based on customer acquisition cost and customer lifetime value, Revenue Analytics based on your raw shop data, Email Segmentation to effectively target customers based on their behavior and finally Holiday Performance Enablers to optimize ad spendings on your key shopping days, ensure stocks are at hand and finally plan potential follow-up strategies to turn holiday buyers into loyal customers.

It’s good to know that Magento is expanding its horizon by acquiring RJMetrics and presenting a viable in-house analytics solution. For those of you who have tested or even implemented RJMetrics in your setups I’d be glad to hear some feedback. For more information about RJMetrics and what Magento Analytics can offer have a look at the RJMetrics website.
Posted on Leave a comment

Magento Newsletter Unsubscribe Form

Magento Logo

One of the most missed functions in Magento 1 is a newsletter unsubscribe form. Although, by default you can generate newsletter unsubscribe links to be used for instance in newsletters sent out to subscribers there’s no nice out-of-the-box way to integrate a newsletter unsubscribe form in Magento 1.

Which is why there already exist some extensions on the Magento Connect Marketplace that extend the existing newsletter functionality by e.g. adding a simple unsubscribe field and toggling the action submitted (“subscribe” vs. “unsubscribe”). Unfortunately, having a look at most of the existing extensions made clear that they either include too much functionality or simply are outdated.

Implementing the Magento Newsletter Unsubscribe Extension

Luckily, implementing the newsletter unsubscribe functionality in Magento is pretty straight forward. In order to share this information here’s the gist on how to create a Magento Newsletter unsubscribe form extension.

Basically, what we are trying to achieve here is an additional Magento page with a distinct URL that incorporates our newsletter unsubscribe form, i.e. “newsletter-unsubscribe”. This form handles submissions and checks if the e-mail submitted currently is subscribed to the Magento newsletter. If that’s the case it will unsubscribe this e-mail based on the built-in Magento functionality, i.e. it will automatically sent out the un-subscription confirmation mail and execute any other events that you might have integrated for newsletter related events. Last but not least, as always everything should be translatable.

Custom route instead of CMS page

Instead of creating an additional CMS page and loading our newsletter unsubscribe form into it (e.g. using layout updates) we are going to create a custom route for our extension. By default, this route will be newsletter-unsubscribe. The relevant entries in the config.xml are as follows:

<?xml version="1.0"?>
<config>
 
  <modules>
    <BothInteract_NewsletterUnsubscribe>
      <version>1.0.1</version>
    </BothInteract_NewsletterUnsubscribe>
  </modules> 
 
  <frontend>
    <routers>
      <bothinteract_newsletterunsubscribe>
        <use>standard</use>
        <args>
          <module>BothInteract_NewsletterUnsubscribe</module>
          <frontName>newsletter-unsubscribe</frontName>
        </args>
      </bothinteract_newsletterunsubscribe>
    </routers>
 
    <secure_url>
      <bothinteract_newsletterunsubscribe>/newsletter-unsubscribe/</bothinteract_newsletterunsubscribe>
    </secure_url>

    ...
</config>

As you can see an additional frontend route is added (newsletter-unsubscribe) that maps to our extensions’s controller.

Display newsletter unsubscribe form

Now that URLs starting with “newsletter-unsubscribe” are routed to our controller we need to make sure that our custom block is loaded and unsubscribe submissions are processed correctly.

As always, the default action is index and in our case it’s responsible for rendering or newsletter unsubscribe form:

public function indexAction() {

  $this->loadLayout();

  // load 1-column page layout
  $this->getLayout()
    ->getBlock('root')
    ->setTemplate('page/1column.phtml');

  // load breadcrumb bar
  $breadcrumbs = $this->getLayout()->getBlock('breadcrumbs');
  $helper = Mage::app()->getHelper('bothinteract_newsletterunsubscribe');

  if ($breadcrumbs) {
    $breadcrumbs->addCrumb('home', array(
      'label' => $helper->__('Home'),
      'title' => $helper->__('Go to Home Page'),
      'link' => Mage::getBaseUrl()));
    $breadcrumbs->addCrumb('newsletter-unsubscribe', array(
      'label' => $helper->__('Newsletter Unsubscribe'),
      'title' => $helper->__('Newsletter Unsubscribe'),
      'link' => Mage::getUrl('newsletter-unsubscribe')));
  }

  // render our custom block
  $block = $this->getLayout()
    ->createBlock('bothinteract_newsletterunsubscribe/customblock')
    ->setTemplate('bothinteract_newsletterunsubscribe/form.phtml');

  // append to CMS content block
  $this->getLayout()->getBlock('content')->append($block);

  $this->renderLayout();
}

There’s no magic here but let’s quickly dissect what is happening:

First, we are loading the layout and setting the 1-column page layout by default .

Hint: Using additional extension backend options we can make these settings available for users to edit directly without touching the code. This will be available in version 1.1.

Since we want our page to appear in the breadcrumbs too we manually add it. Finally, we create and append our custom block while setting the corresponding template and render the final layout.

Process newsletter unsubscribe form submissions

Now that we are able to display the newsletter unsubscribe form it’s time to process form submissions. This is done using the unsubscribe action:

public function unsubscribeAction() {
  if ($this->getRequest()->isPost() && $this->getRequest()->getPost('email')) {
    $session = Mage::getSingleton('core/session');
    $email = (string) $this->getRequest()->getPost('email');

  try {
    if (!Zend_Validate::is($email, 'EmailAddress')) {
      Mage::throwException($this->__('Please enter a valid email address.'));
    }

    $subscriber = Mage::getModel('newsletter/subscriber')
      ->loadByEmail($email);

    if ($subscriber && $subscriber->getId()) {
      // check already unsubscribed/inactivated before
      if ($subscriber->getStatus() == Mage_Newsletter_Model_Subscriber::STATUS_NOT_ACTIVE ||
        $subscriber->getStatus() == Mage_Newsletter_Model_Subscriber::STATUS_UNSUBSCRIBED) {
        $session->addSuccess($this->__('You are not registered with this e-mail.'));
      } else {
        $subscriber->unsubscribe();
        $session->addSuccess($this->__('You have been successfully unsubscribed.'));
      }
    } else {
      $session->addSuccess($this->__('You are not registered with this e-mail.'));
    }
   } catch (Mage_Core_Exception $e) {
     $session->addException($e, $this->__('There was a problem with the unsubscription: %s', $e->getMessage()));
   } catch (Exception $e) {
     $session->addException($e, $this->__('There was a problem with the unsubscription.'));
   }
  }
  $this->_redirectReferer();
}

Again, no magic is happening here. We are simply checking the submitted e-mail, try to load the corresponding newsletter subscriber object and check it’s status. Depending on the status we are able to unsubscribe this e-mail or display the appropriate message.

Our actual custom Magento block is very slim on purpose and only returns the form action URL of our custom extension:

class BothInteract_NewsletterUnsubscribe_Block_Customblock extends Mage_Core_Block_Template {

  public function getFormActionUrl() {
    return $this->getUrl('newsletter-unsubscribe/index/unsubscribe/', array('_secure' => true));
  }
}

Yes, that’s all there is to it. Sorry to disappoint you in case you are expecting magic stuff happening here. Thus, based on Magento’s stable architecture adding a newsletter unsubscribe form can be easily achieved.

Download Magento Newsletter Unsubscribe Extension

This extension has been released by Both Interact and is available for free on the Magento Connect marketplace as Both Interact Newsletter Unsubscribe extension. Feel free to download and rate the extension.

 

Posted on 7 Comments

Fix requested URI error when using owncloud and PHP 7

This is just a quick note for people experiencing problems using ownCloud with PHP 7.0.6 (and potentially versions above) which cause the following error when trying to access the admin backend:

{“reqId”:”qKglxvKVntjfjDJur2Zw”,”remoteAddr”:””,”app”:”index”,”message”:”Exception: {\”Exception\”:\”Exception\”,\”Message\”:\”The requested uri() cannot be processed by the script ‘\\\/apps\\\/owncloud\\\/index.php’)\”,\”Code\”:0,\”Trace\”:\”#0 \\\/apps\\\/owncloud\\\/lib\\\/private\\\/appframework\\\/http\\\/request.php(640): OC\\\\AppFramework\\\\Http\\\\Request->getRawPathInfo()\\n#1 \\\/apps\\\/owncloud\\\/lib\\\/base.php(819): OC\\\\AppFramework\\\\Http\\\\Request->getPathInfo()\\n#2 \\\/apps\\\/owncloud\\\/index.php(39): OC::handleRequest()\\n#3 {main}\”,\”File\”:\”\\\/apps\\\/owncloud\\\/lib\\\/private\\\/appframework\\\/http\\\/request.php\”,\”Line\”:614}”,”level”:3,”time”:”2016-05-07T07:18:43+00:00″}

There’s a simple solution to this problem by patching __isset() in  lib/private/AppFramework/Http/Request.php:

public function __isset($name) {
 + if (in_array($name, $this->allowedKeys, true)) {
 + return true;
 + }
 return isset($this->items['parameters'][$name]);
 }

For more info have a look at the offical GitHub patch commit.

 

Posted on Leave a comment

PHP Mailing List – Version 3.2.0 released

Source Code Icon

Today, version 3.2.0 of PHP Mailing List was released.

Version 3.2.0 of PHP Mailing List is a feature release that was focused on further improving existing anti-spam measures and the usability in the admin interface concerning re-sending membership invitations to pending members.

Short URLs using Google URL shortener API

Since a couple of users reported problems when sending mails to Microsoft based mail services, such as outlook.com and hotmail.com, PHP Mailing List now offers the possibility to automatically shorten action URLs using the Google URL shortener API.

Thus, when activating the URL shortener feature PHP Mailing List will automatically shorten reply-to and authorization links. This way URLs generated by PHP Mailing List will not look as “spammy” as they might do now, especially when using the Google event tracking feature introduced in version 2.0.0.

So, in case you suffer from Microsoft related bounce messages such as

<xyz@hotmail.com>:
Connected to 65.54.188.110 but sender was rejected.
Remote host said: 550 SC-001 (BAY004-MC3F36) Unfortunately, messages from x.x.x.x weren't sent. Please contact your Internet service provider since part of their network is on our block list. You can also refer your provider to http://mail.live.com/mail/troubleshooting.aspx#errors.

make sure to activate to URL shortener feature in PHP Mailing List and re-check your sender score and check for possible listings in various blacklists.

Re-send membership invitation to pending members

In addition, due to numerous requests it is now also possible to re-send membership invitations to already pending members. Simply open up the admin interface and hit the re-send link in the pending invitations screen.

Download PHP Mailing List 3.2.0

As always, feel free to download the latest version of PHP Mailing List via GitHub. Comments and questions are welcome.

Posted on 4 Comments

Sample SEO Magento robots.txt file

Magento Logo

Since I get a lot of requests for a robots.txt file designed for Magento SEO here is a sample to get you started. This Magento robots.txt makes the following assumptions:

  • We don’t differentiate between search engines, hence User-agent: *
  • We allow assets to be crawled
    • i.e. images, CSS and JavaScript files
  • We only allow SEF URLs set in Magento
    • e.g. no direct access to the front controller index.php, view categories and products by ID, etc.
  • We don’t allow filter URLs
    • Please note: The list provided is not complete. In case you have custom extension that use filtering make sure to include these filter URLs and parameters in the filter URLs section.
  • We don’t allow session related URL segments
    • e.g. product comparison, customer, etc.
  • We don’t allow specific files to be crawled
    • e.g. READMEs, cron related files, etc.

Magento robots.txt

Enough of the talking, here comes your SEO Magento robots.txt:

# Crawlers Setup
User-agent: *

# Directories
Disallow: /app/
Disallow: /cgi-bin/
Disallow: /downloader/
Disallow: /includes/
Disallow: /lib/
Disallow: /pkginfo/
Disallow: /report/
Disallow: /shell/
Disallow: /var/

# Paths (clean URLs)
Disallow: /index.php/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /catalogsearch/
#Disallow: /checkout/
Disallow: /control/
Disallow: /contacts/
Disallow: /customer/
Disallow: /customize/
Disallow: /newsletter/
Disallow: /poll/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/
Disallow: /catalog/product/gallery/

# Misc. files you don’t want search engines to crawl
Disallow: /cron.php
Disallow: /cron.sh
Disallow: /composer.json
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /STATUS.txt
Disallow: /mage
#Disallow: /modman
#Disallow: /n98-magerun.phar
Disallow: /scheduler_cron.sh
Disallow: /*.php$

# Disallow filter urls
Disallow: /*?min*
Disallow: /*?max*
Disallow: /*?q*
Disallow: /*?cat*
Disallow: /*?manufacturer_list*
Disallow: /*?tx_indexedsearch

Feel free to leave comments below for additional remarks and suggestions for improvement.

Posted on Leave a comment

Updating Magento tax rules causes code already exists error

Magento Logo

When trying to update tax rules in Magento that have been imported by extensions such as firegento magesetup you might run into problems related to duplicate tax rule codes, e.g. “20% VAT” for both customers and retailers.

By default, Magento happily accepts duplicate tax rule codes when importing them trough setup scripts. But when trying to update these tax rules through adminhtml it will fail due to model checks with a “code already exists” error, as shown below:

Magento tax rule code already exists

The easiest solution here is to manually set the tax rule codes in your table tax_calculation_rule. Simply edit the code column there and you are all set. Happy tax’ing!

Posted on Leave a comment

New table column not updating in Magento

Magento Logo

So you’ve added a custom column for a table, e.g. admin_user or through your custom extension’s model table. The only problem is your values are not being stored by your underlying model classes, i.e. abstractions of Mage_Core_Model_Abstract. But why you ask since Magento is supposed to automatically infer database table to model relations through Zend_Db and magic getters and setters (simply put).

Clear that cache

Well, in fact the solution is pretty simple:

Zend_Db caches the structure of the database tables.

Thus, if you add new columns make sure to refresh the cache. Zend will then update the cache accordingly.

Also note that even if you all your caches are disabled the database structure and other info may still be cached in var/cache, so:

Clear /var/cache.

to make sure that your table adaptions reflect properly.

Posted on Leave a comment

Delete orders in Magento using a shell script

Magento Logo

In a recent post I’ve shown the steps required to delete orders in Magento using a couple of SQL queries. Since this is a common issue when switching from development to staging and finally to production ennvironments it would be handy to have a shell script that takes care of deleting orders in Magento by simply specifying a list of orders to delete, either by order id or order increment id.

One of the problems with installing additional extensions such as Delete Orders in this particular case  as explained in the previous post on deleting orders in Magento is that these extension will overwrite the sales order grid in most of the cases, thus causing conflicts in the code base.

This is why I’ve decided to write up a simply PHP shell script containing the required commands to delete orders in Magento. Although it’s pretty self-explanatory I’m planning to release an extension on Magento Connect. Meanwhile feel free to use the code provided below.

<?php

require '../app/Mage.php';

Mage::app('admin')->setUseSessionInUrl(false);

/**
 * whether $removeOrderIds are specified as order increment ids (e.g. 100000025)
 * or order ids (e.g. 25).
 */
$useOrderIncrementId = true;

/**
 * specify order ids to remove
 */
$removeOrderIds = array(
);

/**
 * Deletes order either by $orderIncrementId or $orderId.
 * @param int $orderId
 * @param boolean $isOrderIncrementId is order increment id
 * @return boolean
 */
function deleteOrder($orderId, $isOrderIncrementId = false) {
 try {

   $order = null;

   // load order either based on order increment id or order id
   if ($isOrderIncrementId) {
     $order = Mage::getModel('sales/order')->load($orderId, 'increment_id');
   } else {
     $order = Mage::getModel('sales/order')->load($orderId);
   }

   if (!$order->getId()) {
     echo 'Order #' . $orderId . ' does not exist (anymore)' . PHP_EOL;
     return;
   }

   // 1. call delete to trigger delete-CASCADE
   $order->delete();

   // 2. manual clean-up for deleted order
   // credits go to DeleteOrders extensions
   $resource = Mage::getSingleton('core/resource');
   $delete = $resource->getConnection('core_read');
   $orderTable = $resource->getTableName('sales_flat_order_grid');
   $invoiceTable = $resource->getTableName('sales_flat_invoice_grid');
   $shipmentTable = $resource->getTableName('sales_flat_shipment_grid');
   $creditMemoTable = $resource->getTableName('sales_flat_creditmemo_grid');
 
   $sql = "DELETE FROM " . $orderTable . " WHERE entity_id = " . $orderId . ";";
   $delete->query($sql);
   $sql = "DELETE FROM " . $invoiceTable . " WHERE order_id = " . $orderId . ";";
   $delete->query($sql);
   $sql = "DELETE FROM " . $shipmentTable . " WHERE order_id = " . $orderId . ";";
   $delete->query($sql);
   $sql = "DELETE FROM " . $creditMemoTable . " WHERE order_id = " . $orderId . ";";
   $delete->query($sql);

   // 3. we are done!
   echo 'Removed order #' . $orderId . PHP_EOL;

   return true;
 } catch (Exception $e) {
   echo "Failed to remove order #" . $orderId . ": "
   . $e->getMessage() . PHP_EOL;
 }

 return false;
}

/**
 * iterate over $removeOrderIds
 */
foreach ($removeOrderIds as $orderId) {
  deleteOrder($orderId, $useOrderIncrementId);
}

echo "DONE!" . PHP_EOL;
Posted on Leave a comment

Please agree to all the terms and conditions before placing the order redirect issue in Paypal Express Checkout and Magento

Magento Logo

So your customers are not able to complete orders when using Paypal Express Checkout in Magento one page checkout due to “misbehaving” sales agreement checkboxes.

Please agree to all the terms and conditions before placing the order.

The scenario at hand is as follows:

  1. Customers select PayPal Express Checkout as checkout method (yes, it’s not a payment method!)
  2. Customers are redirected to the PayPal payment page and select “Buy Now”
  3. Customers are redirected to Magento’s PayPal order review page (/paypal/order/review) to accept your sales agreements set (so far so good)
  4. Customers check all required agreement checkboxes displayed on the order review page and click “Buy now”
  5. But instead of being able to complete their order customers are redirected back to the order review page with a message “Please agree to all the terms and conditions before placing the order” that they need to accept all agreements in order to submit their order – thus ending in a order submission loop and being unable to complete their orders.

Check for required sales agreements

The solution to overcome this problem is rather easy: Check for required agreements via Sales / Agreements that might cause the PayPal order review step block valid order submission.

There’s a chance that some agreements is not rendered (correctly), thus disabling customers from accepting them and actually submitting their order.

A recent customer had the payment option electronic direct debit enabled, based on an third-party extension. Unfortuntalely, this extension did not render its additional required sales agreement correctly leading to the fact that customers were unable to select the corresponding required checkbox on the order review page ending in the above mentioned order submission loop.

Check for required sales agreements at order submission runtime

You can also check at runtime which agreements are required to be able to submit orders by inspecting the Paypal Express Checkout controller: Mage_Paypal_Controller_Express_Abstract. There you find the method placeOrderAction which initially checks for required agreements and their submission value. We can easily inject a debug message there to retrieve the required sales agreements at runtime. Below you find the corresponding code snippet. All we need to do is

Mage::log(implode(', ', $requiredAgreements));

which will write the required agreements to system.log. As always, make sure to enable debug logging in Magento.

public function placeOrderAction()
{
  try {
    $requiredAgreements = Mage::helper('checkout')-&gt;getRequiredAgreementIds();

    Mage::log(implode(', ', $requiredAgreements)); // inject debug logging

    if ($requiredAgreements) {
      $postedAgreements = array_keys($this-&gt;getRequest()-&gt;getPost('agreement', array()));
      if (array_diff($requiredAgreements, $postedAgreements)) {
        Mage::throwException(Mage::helper('paypal')-&gt;__('Please agree to all the terms and conditions before placing the order.'));
      }
    }
...

this will print your required agreement ids in system.log.

Make sure to revert any changes done to this Magento core file!

Posted on 3 Comments

Customize surcharge price logic for Phoenix Cash on Delivery Magento extension

Magento Logo

A recent customer had the requirement to calculate cash on delivery surcharges in relation to the shipping costs. For instance, let’s say that shipping to Germany costs EUR 8 whereas customers from Austria are charged only EUR 5, unless the total order reaches a certain amount resulting in free shipping.

Now the requirement in this case was that shipping costs and cash on delivery surcharges should in total be EUR 12 for shipments to Germany and EUR 8 for shipments to Austria. So the cash on delivery surcharge has to be calculated in relation to the shipping costs selected to match the desired total cash on delivery surcharge + shipping costs.

Using Phoenix Cash on Delivery extension enables us to set fixed or percentage surcharge values depending in inland and foreign shipments. But these settings do not contain special logic for shipping methods and their respective costs. The extension by default only allows us to disable certain shipping methods when using cash on delivery and limit the list of allowed countries, which in general is sufficient enough but not in our particular case.

So, using the vanilla version of the Phoenix Cash on delivery extension would in the example described above result in duplicate costs (shipping + cash on delivery surcharge) for customers during the checkout process, i.e. EUR 8 for shipping + EUR 12 cash on delivery surcharge for German customers, where in fact we only want to charge German customers EUR 12 in total when cash on delivery is selected as payment method and in total EUR 8 for Austrian customers.

Customize Phoenix Cash on Delivery extension

Having a look at the Phoenix Cash on Delivery extension quickly reveals that for our price logic to work we need to customize the price calculcation in Phoenix_CashOnDelivery_Model_CashOnDelivery and especially the functions getInlandCosts and getForeignCountryCosts.

First, copy app/code/community/Phoenix/CashonDelivery/Model/CashOnDelivery.php to app/code/local/Phoenix/CashonDelivery/Model/CashOnDelivery.php to overwrite the model class in the local code pool.

Note that this should be done using a custom module. The example here only serves as demonstration of the required code pieces.

Next, add the function getConditionalCosts in your local CashOnDelivery.php file:

public function getConditionalCost($baseCost) {
  $quote = Mage::getModel('checkout/session')->getQuote();
  if ($quote) {
    if ($quote->getShippingAddress()) {
      // check customer shipping address country
      $shippingData = $quote->getShippingAddress()->getData();

      if (isset($shippingData['country_id'])) {
        if ($shippingData['country_id'] === 'DE') {
          $baseCost; // do some calculation here
        } else if ($shippingData['country_id'] === 'AT') {
          if (floatval($quote->getGrandTotal()) < 70) {
            $baseCost = // do some calculation here
          }
        }
      }
    }
  }

  return floatval($baseCost);
}

And to actually integrate the custom logic by adding the required calls to getConditionalCost in getInlandCosts and getForeignCountryCosts:

public function getInlandCosts($address = null) {

  $inlandCost = $this->getConfigData('inlandcosts');
  $minInlandCost = $this->getConfigData('minimum_inlandcosts');

  if (is_object($address) && Mage::getStoreConfigFlag(self::XML_CONFIG_PATH_CASHONDELIVERY_COST_TYPE)) {
    $calcBase = $this->getConfigData('cost_calc_base');
    $inlandCost = ($address->getData($calcBase) / 100) * $inlandCost;
    if ($inlandCost < $minInlandCost) {
      $inlandCost = $minInlandCost;
    }
  }

  return $this->getConditionalCost($inlandCost);

  //return floatval($inlandCost);
}

public function getForeignCountryCosts($address = null) {
  $foreignCost = $this->getConfigData('foreigncountrycosts');
  $minForeignCost = $this->getConfigData('minimum_foreigncountrycosts');

  if (is_object($address) && Mage::getStoreConfigFlag(self::XML_CONFIG_PATH_CASHONDELIVERY_COST_TYPE)) {
    $calcBase = $this->getConfigData('cost_calc_base');
    $foreignCost = ($address->getData($calcBase) / 100) * $foreignCost;
    if ($foreignCost < $minForeignCost) {
      $foreignCost = $minForeignCost;
    }
  }

  return $this->getConditionalCost($foreignCost);

  //return floatval($foreignCost);
}

Possible optimizations

Note that with a custom module can easily administrate the price values using by getConditionalCost through the admin backend. As always use system.xml to add your backend options. In addition, make sure to rewrite the model through config.xml. But I’ll leave that to you as an additional excercise 😉