<p>Auto Scaling groups have a useful feature for this, named <a href="https://docs.aws.amazon.com/autoscaling/ec2/userguide/lifecycle-hooks.html" rel="nofollow noreferrer">lifecycle hooks</a>.</p>

<p>Worflow taken from the documentation above:</p>

<p><a href="https://i.sstatic.net/1yaNt.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/1yaNt.png" alt="Lifecycle workflow"></a></p>

<p>As you can notice there's a Scale in step, triggering a Terminating:Wait for the autoscaling group and notifying the instance to be terminated, the instance has now to do it's work and once done signal it is ok to be terminated.</p>

<p>If the task can take longer than the autoscaling group HeartbeatTimeout parameter you can reset the timeout with (quoting still from the same page):</p>

<blockquote>
  <p>Restart the timeout period by recording a heartbeat, using the
  <code>record-lifecycle-action-heartbeat</code> command or the
  <code>RecordLifecycleActionHeartbeat</code> operation. This increments the
  heartbeat timeout by the timeout value specified when you created the
  lifecycle hook. For example, if the timeout value is 1 hour, and you
  call this command after 30 minutes, the instance remains in a wait
  state for an additional hour, or a total of 90 minutes.</p>
</blockquote>

<p>So for your case, the lifecycle notification should startup a kind of script/program which will:</p>

<ul>
<li>prevent this builder to receive new builds</li>
<li>loop periodically to check if the ongoing builds are done</li>
<li>if there's build still in progress and the timeout is near, reset the timer</li>
<li>if there's no more builds in progress, signal to proceed to termination</li>
</ul>

CC BY-SA 4.0
2018-05-28T15:04:30.667Z
2018-05-28T15:04:30.667Z
Tensibai
<blockquote>
  <p>My concern is that when scaling down, the ASG may terminate instances
  that are in the middle of a build.</p>
</blockquote>

<p>It the builds are running then one should prevent down scaling. When the build has been completed then one could down scale as the resources will not be required anymore.</p>

CC BY-SA 3.0
2017-10-29T22:55:53.703Z
Can you elaborate on
>one should prevent down scaling
?

Are you suggesting that I should disable scaling while builds are running on any agents? This would result in quite a lot of idle agents at 4:00pm when only a few builds are running. What mechanism would I use to trigger the disabling of scaling?

Also, GoCD agents run one build per VM with a fairly long spin up time, so I prefer to have at least a small buffer of idle instances.
user2640621
CC BY-SA 3.0
2017-10-29T12:45:01.207Z
2017-10-29T12:45:01.207Z
030
<p>I think we have an answer here. Using GOCD API it can be managed. <a href="https://amaysim.engineering/auto-scaling-build-agents-for-gocdtags-a10f12d5b77c" rel="nofollow noreferrer">https://amaysim.engineering/auto-scaling-build-agents-for-gocdtags-a10f12d5b77c</a></p>

CC BY-SA 4.0
2018-05-28T14:02:38.240Z
Welcome DevOps! Please note that link-only answers are considered very low quality (the links can go dead). Adding a short summary or a quote of the relevant info from the referenced content in the post itself is highly recommended.
Dan Cornilescu
CC BY-SA 4.0
2018-05-28T11:23:49.093Z
2018-05-28T11:23:49.093Z
Anirban
<p>At the moment I have an AutoScaling Group (ASG) of GoCD build agents without any scaling policies. I have created some custom metrics that indicate how many build agents are currently idle and I'd like to scale based off of that. My concern is that when scaling down, the ASG may terminate instances that are in the middle of a build. This may result in failed builds and delayed builds.</p>

<p>How can I scale down an ASG without terminating instances that are in use?</p>

CC BY-SA 3.0
2017-10-29T12:43:25.990Z
Did you try to scale down? If true, what was the outcome?
030
CC BY-SA 3.0
2017-10-29T22:52:25.400Z
I have not. I can't risk terminating real builds in production, and I unfortunately don't have a good environment to test with. However, I know that the ASG will randomly pick instances to terminate (how would it know which instances are running builds?) so I'm quite confident that this is a real issue. I place a lot of emphasis on not risking terminating builds because to me, build/release speed is one of the most important things to our engineering department.
user2640621
<continuous-integration><amazon-ec2><continuous-deployment><autoscaling>
Autoscaling on GoCD agents without terminating active builds?
user2640621
# Autoscaling on GoCD agents without terminating active builds?

(Asked by: username_0 on 2017-10-21)

At the moment I have an AutoScaling Group (ASG) of GoCD build agents without any scaling policies. I have created some custom metrics that indicate how many build agents are currently idle and I'd like to scale based off of that. My concern is that when scaling down, the ASG may terminate instances that are in the middle of a build. This may result in failed builds and delayed builds.

How can I scale down an ASG without terminating instances that are in use?

**Tags:** continuous-integration, amazon-ec2, continuous-deployment, autoscaling

> **Comments:**
> **username_1**: Did you try to scale down? If true, what was the outcome?  
> **username_0**: I have not. I can't risk terminating real builds in production, and I unfortunately don't have a good environment to test with. However, I know that the ASG will randomly pick instances to terminate (how would it know which instances are running builds?) so I'm quite confident that this is a real issue. I place a lot of emphasis on not risking terminating builds because to me, build/release speed is one of the most important things to our engineering department.  

---

## 3 answers

### Answer by username_2 on 2018-05-28. Score: 3

Auto Scaling groups have a useful feature for this, named [lifecycle hooks](https://docs.aws.amazon.com/autoscaling/ec2/userguide/lifecycle-hooks.html).

Worflow taken from the documentation above:

[![Lifecycle workflow](https://i.sstatic.net/1yaNt.png)](https://i.sstatic.net/1yaNt.png)

As you can notice there's a Scale in step, triggering a Terminating:Wait for the autoscaling group and notifying the instance to be terminated, the instance has now to do it's work and once done signal it is ok to be terminated.

If the task can take longer than the autoscaling group HeartbeatTimeout parameter you can reset the timeout with (quoting still from the same page):

> Restart the timeout period by recording a heartbeat, using the
> `record-lifecycle-action-heartbeat` command or the
> `RecordLifecycleActionHeartbeat` operation. This increments the
> heartbeat timeout by the timeout value specified when you created the
> lifecycle hook. For example, if the timeout value is 1 hour, and you
> call this command after 30 minutes, the instance remains in a wait
> state for an additional hour, or a total of 90 minutes.

So for your case, the lifecycle notification should startup a kind of script/program which will:

* prevent this builder to receive new builds
* loop periodically to check if the ongoing builds are done
* if there's build still in progress and the timeout is near, reset the timer
* if there's no more builds in progress, signal to proceed to termination


---

### Answer by username_1 on 2017-10-29. Score: 2

> My concern is that when scaling down, the ASG may terminate instances
> that are in the middle of a build.

It the builds are running then one should prevent down scaling. When the build has been completed then one could down scale as the resources will not be required anymore.

> **Comments:**
> **username_0**: Can you elaborate on >one should prevent down scaling ?  Are you suggesting that I should disable scaling while builds are running on any agents? This would result in quite a lot of idle agents at 4:00pm when only a few builds are running. What mechanism would I use to trigger the disabling of scaling?  Also, GoCD agents run one build per VM with a fairly long spin up time, so I prefer to have at least a small buffer of idle instances.  

---

### Answer by username_3 on 2018-05-28. Score: 0

I think we have an answer here. Using GOCD API it can be managed. <https://amaysim.engineering/auto-scaling-build-agents-for-gocdtags-a10f12d5b77c>

> **Comments:**
> **username_4**: Welcome DevOps! Please note that link-only answers are considered very low quality (the links can go dead). Adding a short summary or a quote of the relevant info from the referenced content in the post itself is highly recommended.  

---

