<p>AWS have a specific service to deal with this type of use case, <a href="https://aws.amazon.com/systems-manager/" rel="nofollow noreferrer">EC2 Systems Manager</a></p>

<p>This allows making base patch sets and then checking they are applied to target systems. </p>

<blockquote>
  <p>"Patch Manager is a fully-managed service (provided at no additional
  cost) that helps you simplify your operating system patching process,
  including defining the patches you want to approve for deployment, the
  method of patch deployment, the timing for patch roll-outs, and
  determining patch compliance status across your entire fleet of
  instances. It’s extremely configurable with some sensible defaults and
  helps you easily deal with patching heterogeneous clusters."</p>
</blockquote>

<p>-- <a href="https://aws.amazon.com/blogs/aws/amazon-ec2-systems-manager-patch-manager-now-supports-linux/" rel="nofollow noreferrer">"Amazon EC2 Systems Manager Patch Manager now supports Linux"</a></p>

<p>You'll need to install an SSM agent on each system and add a role but other than that it should be unobtrusive </p>

CC BY-SA 3.0
2018-04-08T04:33:07.953Z
looks good but this solution is not including ELB/ALB disconnect/connect orchestration while applying patches
rombob
CC BY-SA 3.0
2018-04-08T18:05:03.707Z
The AWS solution to your problem is to use an immutable solution: build AMI with the new kernel and then use ASG to cycle them out of ALB but I know you don't want this!!
Vorsprung
CC BY-SA 3.0
2018-04-08T19:09:42.273Z
If I had to solve this problem I'd be less concerned about rotating them out before the update.  Apart from increased IO a kernel update will have no effect until the reboot.  You could have a init.d (or upstart or systemd or whatever) script to make a controlled exit from the ALB group on a controlled rebooted
Vorsprung
CC BY-SA 3.0
2018-04-06T12:39:28.930Z
2018-04-06T12:39:28.930Z
Vorsprung
<p>As people in the comments are saying, you can do this with aws-cli in bash or in the python boto3 library. </p>

<p>What about ansible? It has the ability to see your ec2 instances and it could handle the automation for maintenance tasks. If you are not using ansible I highly recommend you add that tool to your DevOps toolbox. </p>

<p>Here is how you might structure your <a href="https://stackoverflow.com/questions/27315469/ansible-how-to-sequentially-execute-playbook-for-each-host">playbook</a> to do what you are wanting.</p>

CC BY-SA 3.0
2018-04-04T17:32:08.120Z
2018-04-04T17:32:08.120Z
Levi
<p>Would like to ask for recommendations on ready automated solutions or templates for rolling ec2 instance upgrade on AWS, requiring a reboot with ELB/ALB.
Specifically required to perform the maintenance upgrade for the kernel on hundreds of Linux machines.</p>

<p>For each ELB/ALB steps are:</p>

<ol>
<li>Disconnect ec2 instance from ELB/ALB wait for connection draining</li>
<li>Upgrade kernel and reboot machine (not part of automation, e.g. running external ./upgrade.sh)</li>
<li>Wait for instance to be online, connect to ELB/ALB, wait for health check status </li>
<li>Continue to next instance in the load balancer</li>
</ol>

<p>Note: not looking for migration to the solution similar to Netflix Spinnaker or immutable deployment strategy implementation, but for general and simple on-demand maintenance tool.</p>

CC BY-SA 3.0
2018-03-27T15:21:47.327Z
Is doing a +1 deploy possible for you? So instead of removing an instance from the ALB, you *add* an already upgraded instance? This would save a bunch of time from the two _waiting_ states you listed.
MrDuk
CC BY-SA 3.0
2018-03-27T18:01:35.953Z
Thank you @MrDuk for the suggestion, this is what was mentioned as "immutable deployment" which will require currently more resources and coordination for deploying tens of different applications on hundreds of servers with different configurations, ideally this would be possible with better deployment automation, but it's not the case currently
rombob
CC BY-SA 3.0
2018-03-27T18:13:25.863Z
If each node is distinct, with custom changes to each, I don't know that a tool exists that can help you with this. However, you could script this fairly easily using the aws-cli, the most complex part being the upgrade/reboot/detecting
MrDuk
CC BY-SA 3.0
2018-03-27T18:16:51.780Z
@rombob I have to agree with MrDuk. If your instances have the capability to run different versions in parallel you should be able to use an immutable deployment strategy with no more effort than the steps you proposed.
Preston Martin
CC BY-SA 3.0
2018-03-27T18:22:30.283Z
@MrDuk agreed, aws-cli or aws-sdk is what make sense to use, wondering if there ready solutions, other than custom upgrade, sequential/parallel traversing of balanced instances seems like very common task before we jump in writing our own, worth checking with community
rombob
CC BY-SA 3.0
2018-03-27T18:28:50.237Z
@PrestonM I agree with with you both :) but it's not an option currently to re-engineer deployment strategy especially for legacy apps, and yet, we need production kernels to have latest patches
rombob
CC BY-SA 3.0
2018-03-27T18:48:30.883Z
Is step 2 (including the application starting when it reboots) already automated?
MrDuk
CC BY-SA 3.0
2018-03-28T11:09:28.360Z
clarified step 2, yes, it's already automated
rombob
CC BY-SA 4.0
2018-05-24T07:13:23.637Z
This could be (somewhat) easily achieved by 1) updating the AMI to get kernel updates, 2) update the ASG's launch config to point to the new AMI and 3) terminate an instance and let the ASG replace it with the new AMI. This is what we're doing for stateless services. For stateful services, we add a volume snapshot/restore or remount the data volumes to the new instances.
SaxDaddy
<amazon-web-services><amazon-ec2><automation><linux><aws-elb>
How to automate rolling upgrades for ec2 running linux behind elastic and/or application load balancer?
rombob
# How to automate rolling upgrades for ec2 running linux behind elastic and/or application load balancer?

(Asked by: username_0 on 2018-03-27)

Would like to ask for recommendations on ready automated solutions or templates for rolling ec2 instance upgrade on AWS, requiring a reboot with ELB/ALB.
Specifically required to perform the maintenance upgrade for the kernel on hundreds of Linux machines.

For each ELB/ALB steps are:

1. Disconnect ec2 instance from ELB/ALB wait for connection draining
2. Upgrade kernel and reboot machine (not part of automation, e.g. running external ./upgrade.sh)
3. Wait for instance to be online, connect to ELB/ALB, wait for health check status
4. Continue to next instance in the load balancer

Note: not looking for migration to the solution similar to Netflix Spinnaker or immutable deployment strategy implementation, but for general and simple on-demand maintenance tool.

**Tags:** amazon-web-services, amazon-ec2, automation, linux, aws-elb

> **Comments:**
> **username_1**: Is doing a +1 deploy possible for you? So instead of removing an instance from the ALB, you *add* an already upgraded instance? This would save a bunch of time from the two _waiting_ states you listed.  
> **username_0**: Thank you @username_1 for the suggestion, this is what was mentioned as "immutable deployment" which will require currently more resources and coordination for deploying tens of different applications on hundreds of servers with different configurations, ideally this would be possible with better deployment automation, but it's not the case currently  
> **username_1**: If each node is distinct, with custom changes to each, I don't know that a tool exists that can help you with this. However, you could script this fairly easily using the aws-cli, the most complex part being the upgrade/reboot/detecting  
> **username_2**: @username_0 I have to agree with username_1. If your instances have the capability to run different versions in parallel you should be able to use an immutable deployment strategy with no more effort than the steps you proposed.  
> **username_0**: @username_1 agreed, aws-cli or aws-sdk is what make sense to use, wondering if there ready solutions, other than custom upgrade, sequential/parallel traversing of balanced instances seems like very common task before we jump in writing our own, worth checking with community  
> **username_0**: @PrestonM I agree with with you both :) but it's not an option currently to re-engineer deployment strategy especially for legacy apps, and yet, we need production kernels to have latest patches  
> **username_1**: Is step 2 (including the application starting when it reboots) already automated?  
> **username_0**: clarified step 2, yes, it's already automated  
> **username_3**: This could be (somewhat) easily achieved by 1) updating the AMI to get kernel updates, 2) update the ASG's launch config to point to the new AMI and 3) terminate an instance and let the ASG replace it with the new AMI. This is what we're doing for stateless services. For stateful services, we add a volume snapshot/restore or remount the data volumes to the new instances.  

---

## 2 answers

### Answer by username_4 on 2018-04-06. Score: 2 | Accepted answer

AWS have a specific service to deal with this type of use case, [EC2 Systems Manager](https://aws.amazon.com/systems-manager/)

This allows making base patch sets and then checking they are applied to target systems.

> "Patch Manager is a fully-managed service (provided at no additional
> cost) that helps you simplify your operating system patching process,
> including defining the patches you want to approve for deployment, the
> method of patch deployment, the timing for patch roll-outs, and
> determining patch compliance status across your entire fleet of
> instances. It’s extremely configurable with some sensible defaults and
> helps you easily deal with patching heterogeneous clusters."

-- ["Amazon EC2 Systems Manager Patch Manager now supports Linux"](https://aws.amazon.com/blogs/aws/amazon-ec2-systems-manager-patch-manager-now-supports-linux/)

You'll need to install an SSM agent on each system and add a role but other than that it should be unobtrusive

> **Comments:**
> **username_0**: looks good but this solution is not including ELB/ALB disconnect/connect orchestration while applying patches  
> **username_4**: The AWS solution to your problem is to use an immutable solution: build AMI with the new kernel and then use ASG to cycle them out of ALB but I know you don't want this!!  
> **username_4**: If I had to solve this problem I'd be less concerned about rotating them out before the update.  Apart from increased IO a kernel update will have no effect until the reboot.  You could have a init.d (or upstart or systemd or whatever) script to make a controlled exit from the ALB group on a controlled rebooted  

---

### Answer by username_5 on 2018-04-04. Score: 1

As people in the comments are saying, you can do this with aws-cli in bash or in the python boto3 library.

What about ansible? It has the ability to see your ec2 instances and it could handle the automation for maintenance tasks. If you are not using ansible I highly recommend you add that tool to your DevOps toolbox.

Here is how you might structure your [playbook](https://stackoverflow.com/questions/27315469/ansible-how-to-sequentially-execute-playbook-for-each-host) to do what you are wanting.


---

