Andrew Cavanagh

Developer.  Project Architect.  Huge Nerd.

(not necessarily in that order)

I frequently use a microservice architecture when building applications. This leads to having a bunch of workers for different tasks. Often these workers will need to autoscale based on how heavily they’re being used. That all works fine, but what I’ve found is that when aws excecutes your scale-in policy it doesn’t actually check to see if the workers are in the middle of something before it kills them. This can result in problems if the worker was part way through processing a long running job when it got the terminate signal. It’s possible to protect them from scale-in termination using the aws sdk. This allows you to mark the instance as protected when it takes a job off the queue, and then release it’s protection when the job finishes or fails. You can see an example of this in this trait:

use Aws\AutoScaling\AutoScalingClient;
use Illuminate\Support\Facades\Log;

trait AwsAutoscalingTrait
    private function setInstanceProtection($state = false)
        try {
            $instance_id             = file_get_contents("http://instance-data/latest/meta-data/instance-id");
            $autoscaling_group = $MY AUTOSCALING GROUP;
            $ec2Client               = $this->getAutoScalingClient();
                'AutoScalingGroupName' => $autoscaling_group,
                'InstanceIds'                     => [$instance_id],
                'ProtectedFromScaleIn'   => $state
        } catch (\Exception $e) {
            Log::debug('You apparently aren\'t on an ec2, so nothing to do here');
    private function getAutoScalingClient()
        $client = new AutoScalingClient([
            'key'      => env('AWS_ACCESS_KEY'),
            'secret'  => env('AWS_SECRET_KEY'),
            'region'  => env('AWS_REGION'),
            'version' => '2011-01-01'
        return $client;

This allows you to just call $this->setInstanceProtection(true); when the worker pulls a job off the queue, and then $this->setInstanceProtection(false); as soon as the job finishes or fails.

comments powered by Disqus