<h1>Autoscaling is a good case for machine learning</h1>

<h2>This is a hard problem to do well.</h2>

<p>What you really want is something like Nest Thermostat for your EC2 infrastructure.</p>

<h2>There are (aforementioned) multiple dimensions of resource demand/limitations.</h2>

<ul>
<li>CPU</li>
<li>memory</li>
<li>disk space</li>
<li>disk IO</li>
<li>network IO</li>
<li>concurrency/latency/queue depth</li>
</ul>

<h2>There are multiple indicators of demand (in addition to the above).</h2>

<ul>
<li>concurrent unique visits/sessions</li>
<li>pages per visit/session</li>
<li>rate of engagement/interaction feature usage</li>
</ul>

<h2>There are multiple common patterns of changing demand over time.</h2>

<ul>
<li>daily user demand cycles</li>
<li>weekly user demand cycles</li>
<li>monthly ...</li>
<li>annual ...</li>
<li>special event/days</li>
<li>DDoS load</li>
<li>media/marketing exposure traffic spikes</li>
</ul>

<h2>There are multiple financial decision factors.</h2>

<ul>
<li>does revenue scale with traffic? How? (How conservative do you need to be?)</li>
<li>is there a hidden cost to control (transaction costs, limits)?</li>
<li>what's the cost model of scaling? (things in a pipeline scale together, things in a load-balanced cluster scale independently)</li>
</ul>

<p>Before long, if you try to hand-optimize on hand-selected features you're going to have a monster of technical debt that is possibly more complicated than any other logic in your site. Amazon makes more money when you err (with a large margin) on the side of caution, so their tools will probably never get close to what you want.</p>

<p>Instead, choose an architecture/technology stack that can grow/scale so you don't have to get it exactly right the first time. Then pick a few factors which you think are obvious. Then try to come up with a way to sort multiple representative possibilities in order of preference. Then collect some real world data covering all those points. If you're lucky, a simple obvious hand-coded solution will jump out at you from looking at the data. If not, code up something that will give you an approximate model f(x1,x2,x3,x4) --> y * app nodes, using an appropriate algorithm.</p>

<p>I bet you didn't think this one was going to be so much fun!</p>

CC BY-SA 3.0
2017-10-27T22:37:08.523Z
I don't think this answer is appropriate. It's not clear from the title (and perhaps it should be edited) but the question specifically asks about scaling Docker instances. There are only two resources in contention on an instance that hosts Docker containers - allocated CPU, and allocated memory. All other resources (including all the ones you've listed) are in contention on the _containers_. Your answer is valid for EC2-based applications, but doesn't apply to Docker.
user2640621
CC BY-SA 3.0
2017-10-29T20:01:35.153Z
I disagree with your interpretation of the OP question. Docker is one example, but the author seems to want help coming up with a broad and general solution, which renders scale targets in EC2 units.
Jeremy
CC BY-SA 3.0
2017-10-24T03:29:40.640Z
2017-10-24T03:29:40.640Z
Jeremy
<p>To be able to scale in with confidence on ECS, there's a few strategies:</p>

<ul>
<li>Create a 3rd custom metric which tracks the greater of the two metrics. For example, if CPU allocation is at 60% and memory allocation is at 70%, the metric should be set to 70%.</li>
<li>Choose one of the two resources (CPU or memory), and always allocate a higher percentage of it to each container compared to the other resource. This way, you will always reach a shortage of your preferred resource before the other, and you don't have to worry about scaling up or down on it. While this is the easiest solution, the obvious downside is that you can't optimise for services that may be CPU or memory intensive, and you may end up with wasted resources.</li>
<li>Do autoscaling without using CloudWatch metrics. Tools such as <a href="https://github.com/ameir/ECSpander" rel="nofollow noreferrer">https://github.com/ameir/ECSpander</a> are available to assist with this.</li>
</ul>

<p>Kubernetes doesn't use CloudWatch metrics for scaling, it manages scaling by setting the ASG desired number of instances through internal mechanisms and therefore isn't subject to the issue you've described.</p>

CC BY-SA 3.0
2017-10-21T03:02:36.700Z
2017-10-27T22:38:30.740Z
2017-10-27T22:38:30.740Z
user2640621
<p>Cloud environments in AWS allow for multi-tenancy managed by the user himself, classic example are container orchestrators such as ECS or Kubernetes.</p>

<p>When you have two services, one needs memory another cpu and you put these in a single cluster. Then scaling-up is relatively trivial. Each time you need more capacity in terms of <em>either</em> cpu <em>or</em> memory, add more capacity. Since EC2 capacity means units in cpu and memory both.</p>

<p><em>Scaling up</em> based on a single metric can very easily achieved using CloudWatch Alarms.</p>

<p>When <em>scaling down</em>, in order to reduce cost it requires to take into account both memory and cpu limits and not let any of the two drop below the required amount.</p>

<p>Since unfortunately CloudWatch Alarms do not allow to use boolean logic or take into account multiple metrics.</p>

<p><strong>What is a good way to implement <em>scale down</em> of capacity for an auto scaling group?</strong></p>

CC BY-SA 3.0
2017-10-29T23:35:21.353Z
Can you please clarify whether your question refers to scaling ASGs in general or specifically scaling Docker instances?
user2640621
CC BY-SA 3.0
2017-10-30T16:51:00.883Z
When I write “Auto Scaling Group” I mean the EC2 service called ASG, not Docker containers.
Evgeny Zislis
CC BY-SA 3.0
2017-10-30T20:57:20.653Z
It's still not clear whether you're referring to scaling EC2 instances that host Docker containers specifically, or EC2 instances in general including traditionally hosted applications that don't use Docker.
user2640621
CC BY-SA 3.0
2017-11-01T09:27:27.837Z
It doesn’t really matter, as long as there is some multi-tenancy in place. For example a Mesos cluster running non-Docker framework is also applicable in this question.
Evgeny Zislis
<amazon-ec2><amazon-cloudwatch>
How to scale-down in a multi-tenant environment?
Evgeny Zislis
# How to scale-down in a multi-tenant environment?

(Asked by: username_0 on 2017-06-22)

Cloud environments in AWS allow for multi-tenancy managed by the user himself, classic example are container orchestrators such as ECS or Kubernetes.

When you have two services, one needs memory another cpu and you put these in a single cluster. Then scaling-up is relatively trivial. Each time you need more capacity in terms of *either* cpu *or* memory, add more capacity. Since EC2 capacity means units in cpu and memory both.

*Scaling up* based on a single metric can very easily achieved using CloudWatch Alarms.

When *scaling down*, in order to reduce cost it requires to take into account both memory and cpu limits and not let any of the two drop below the required amount.

Since unfortunately CloudWatch Alarms do not allow to use boolean logic or take into account multiple metrics.

**What is a good way to implement *scale down* of capacity for an auto scaling group?**

**Tags:** amazon-ec2, amazon-cloudwatch

> **Comments:**
> **username_1**: Can you please clarify whether your question refers to scaling ASGs in general or specifically scaling Docker instances?  
> **username_0**: When I write “Auto Scaling Group” I mean the EC2 service called ASG, not Docker containers.  
> **username_1**: It's still not clear whether you're referring to scaling EC2 instances that host Docker containers specifically, or EC2 instances in general including traditionally hosted applications that don't use Docker.  
> **username_0**: It doesn’t really matter, as long as there is some multi-tenancy in place. For example a Mesos cluster running non-Docker framework is also applicable in this question.  

---

## 2 answers

### Answer by username_2 on 2017-10-24. Score: 2

# Autoscaling is a good case for machine learning

## This is a hard problem to do well.

What you really want is something like Nest Thermostat for your EC2 infrastructure.

## There are (aforementioned) multiple dimensions of resource demand/limitations.

* CPU
* memory
* disk space
* disk IO
* network IO
* concurrency/latency/queue depth

## There are multiple indicators of demand (in addition to the above).

* concurrent unique visits/sessions
* pages per visit/session
* rate of engagement/interaction feature usage

## There are multiple common patterns of changing demand over time.

* daily user demand cycles
* weekly user demand cycles
* monthly ...
* annual ...
* special event/days
* DDoS load
* media/marketing exposure traffic spikes

## There are multiple financial decision factors.

* does revenue scale with traffic? How? (How conservative do you need to be?)
* is there a hidden cost to control (transaction costs, limits)?
* what's the cost model of scaling? (things in a pipeline scale together, things in a load-balanced cluster scale independently)

Before long, if you try to hand-optimize on hand-selected features you're going to have a monster of technical debt that is possibly more complicated than any other logic in your site. Amazon makes more money when you err (with a large margin) on the side of caution, so their tools will probably never get close to what you want.

Instead, choose an architecture/technology stack that can grow/scale so you don't have to get it exactly right the first time. Then pick a few factors which you think are obvious. Then try to come up with a way to sort multiple representative possibilities in order of preference. Then collect some real world data covering all those points. If you're lucky, a simple obvious hand-coded solution will jump out at you from looking at the data. If not, code up something that will give you an approximate model f(x1,x2,x3,x4) --> y \* app nodes, using an appropriate algorithm.

I bet you didn't think this one was going to be so much fun!

> **Comments:**
> **username_1**: I don't think this answer is appropriate. It's not clear from the title (and perhaps it should be edited) but the question specifically asks about scaling Docker instances. There are only two resources in contention on an instance that hosts Docker containers - allocated CPU, and allocated memory. All other resources (including all the ones you've listed) are in contention on the _containers_. Your answer is valid for EC2-based applications, but doesn't apply to Docker.  
> **username_2**: I disagree with your interpretation of the OP question. Docker is one example, but the author seems to want help coming up with a broad and general solution, which renders scale targets in EC2 units.  

---

### Answer by username_1 on 2017-10-21. Score: 1

To be able to scale in with confidence on ECS, there's a few strategies:

* Create a 3rd custom metric which tracks the greater of the two metrics. For example, if CPU allocation is at 60% and memory allocation is at 70%, the metric should be set to 70%.
* Choose one of the two resources (CPU or memory), and always allocate a higher percentage of it to each container compared to the other resource. This way, you will always reach a shortage of your preferred resource before the other, and you don't have to worry about scaling up or down on it. While this is the easiest solution, the obvious downside is that you can't optimise for services that may be CPU or memory intensive, and you may end up with wasted resources.
* Do autoscaling without using CloudWatch metrics. Tools such as <https://github.com/ameir/ECSpander> are available to assist with this.

Kubernetes doesn't use CloudWatch metrics for scaling, it manages scaling by setting the ASG desired number of instances through internal mechanisms and therefore isn't subject to the issue you've described.


---

