503 Error Back-end server is at capacity AWS ELB

When receiving Back-end server is at capacity 503 errors from your Elastic Load Balancer (ELB) instance in the Amazon Web Services (AWS) EC2 be sure to check your HealthyHostCount monitoring setup in the CloudWatch. ELB instances will return 503 errors (apart from other conditions) when your HealthyHostCount monitoring setup has reached the threshold set and stop any further processing, while the backend service might still work correctly which might cause you some unexpected headache.

Thus, check that the file/path you’ve set (e.g. /ping.php) can be reached by the ELB-HealthChecker and that the HTTP code returned is 200. So, for instance redirects to the monitoring file (e.g. /ping.php) will be treated as an error by HealthChecker (HTTP status != 200). In order to check if your configured health monitoring works (HTTP status == 200) as usual check your server logs for ELB-HealthChecker entries:

grep ELB-HealthChecker /var/log/apache2/*

x.y.z.1 - - [28/Aug/2014:12:38:01 +0200] "GET /ping.php HTTP/1.1" 301 464 "-" "ELB-HealthChecker/1.0"
x.y.z.2 - - [28/Aug/2014:12:38:11 +0200] "GET /ping.php HTTP/1.1" 301 464 "-" "ELB-HealthChecker/1.0"
x.y.z.1 - - [28/Aug/2014:12:38:31 +0200] "GET /ping.php HTTP/1.1" 301 448 "-" "ELB-HealthChecker/1.0"
x.y.z.2 - - [28/Aug/2014:12:38:41 +0200] "GET /ping.php HTTP/1.1" 200 183 "-" "ELB-HealthChecker/1.0"
x.y.z.1 - - [28/Aug/2014:12:39:01 +0200] "GET /ping.php HTTP/1.1" 200 183 "-" "ELB-HealthChecker/1.0"
x.y.z.2 - - [28/Aug/2014:12:39:11 +0200] "GET /ping.php HTTP/1.1" 200 239 "-" "ELB-HealthChecker/1.0"

As you can see here a redirect caused a 301 HTTP status to be returned for the first three entries in the list, which automatically incremented the UnHealthyHostCount in CloudWatch and reduced the HealthyHostCount. For further info visit the ELB Developer Guide.