Andrew Cavanagh

Developer.  Project Architect.  Huge Nerd.

(not necessarily in that order)

We had an issue with one of our AWS autoscaling workers recently. Our builds are set up so that CodeDeploy runs a deployment on the new EC2 when it’s launched by the autoscaling group. I think that what happened is that we made a change to the CodeDeploy application via CloudFormation while a deployment was running, and left stranded LifeCycle hooks. This was causing all new instances launched by the autoscaling group to fail with a Heartbeat Timeout error. These docs and this post helped me to figure out and fix the issue. The short version is that I disconnected the CodeDeploy application from the AutoscalingGroup so that deployments would stop failing, then got the orphaned hook with this command:

aws autoscaling describe-lifecycle-hooks --auto-scaling-group-name $MyAutoScalingGroupName

and deleted it with this command:

aws autoscaling delete-lifecycle-hook --auto-scaling-group-name MyAutoScalingGroupName --lifecycle-hook-name $LifeCycleHookNameFromAboveCommandResult

We no longer receive the Heartbeat Timeout error and workers are scaling and deploying properly.

comments powered by Disqus