<p>CloudWatch is absolutely <a href="https://blog.developer.bazaarvoice.com/2015/04/17/predictively-scaling-ec2-instances-with-custom-cloudwatch-metrics/" rel="noreferrer">the way to go</a> on this front, you can surface Kafka Metrics in CloudWatch either by:</p>

<ol>
<li>Having a <a href="https://github.com/sidcarter/kafka-cloudwatch" rel="noreferrer">separate process</a> that pushes the metrics into CloudWatch.</li>
<li>Have your Producers, Consumers or Stream Processors push the metrics you need into CloudWatch.</li>
</ol>

<p>The process you follow will broadly be broken down into:</p>

<ol>
<li><a href="http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/publishingMetrics.html" rel="noreferrer">Publish Custom Metrics</a>.</li>
<li>Create an <a href="http://docs.aws.amazon.com/autoscaling/latest/userguide/GettingStartedTutorial.html#gs-create-lc" rel="noreferrer">AutoScaling Launch Configuration</a>.</li>
<li>Create an <a href="http://docs.aws.amazon.com/autoscaling/latest/userguide/GettingStartedTutorial.html#gs-create-asg" rel="noreferrer">AutoScaling Group and Policies</a> based upon your Custom Metric.</li>
</ol>

<p>Depending on your cost model I would also consider how you can use spot instances to keep the costs down, i.e. if you have non-critical but high compute workloads use <a href="https://aws.amazon.com/ec2/spot/" rel="noreferrer">Spot Instances</a> as part of your autoscaling configuration.</p>

CC BY-SA 3.0
2017-11-08T17:30:57.123Z
The first link is dead
Marquis Blount
CC BY-SA 3.0
2017-12-14T00:12:19.073Z
@MarquisBlount here's the [Google Cache for the page](https://webcache.googleusercontent.com/search?q=cache:mM9WvsLriCQJ:https://blog.developer.bazaarvoice.com/2015/04/17/predictively-scaling-ec2-instances-with-custom-cloudwatch-metrics/)
dededecline
CC BY-SA 3.0
2017-04-04T14:09:47.813Z
2017-04-04T14:09:47.813Z
Richard Slater
<p>I have autoscaling groups that I want to dynamically scale up/down based on the amount of lag showing in certain Kafka topics. There's a 1:1 relationship between topic and autoscaling group that needs to be scaled. I'm fairly sure I can't use CloudWatch metrics to approach this.</p>

<p>Is there a way I could accomplish this using Jenkins without creating race conditions and other issues?</p>

<p>Note: My application is not constrained by CPU, Disk I/O, or any other metric provided by CloudWatch; the constraint is the application's ability to process from incoming Kafka topics.</p>

CC BY-SA 3.0
2017-04-04T12:53:19.453Z
Maybe you can explain a little more why you think cloudwatch is not a good approach ?
Tensibai
CC BY-SA 3.0
2017-04-04T14:05:09.723Z
My application is not constrained by CPU, Disk I/O, or any other metric provided by CloudWatch; the constraint is the application's ability to process from incoming Kafka topics.
dededecline
<jenkins><amazon-web-services><autoscaling>
How can I autoscale instances based on Kafka topic lag?
dededecline
# How can I autoscale instances based on Kafka topic lag?

(Asked by: username_0 on 2017-04-03)

I have autoscaling groups that I want to dynamically scale up/down based on the amount of lag showing in certain Kafka topics. There's a 1:1 relationship between topic and autoscaling group that needs to be scaled. I'm fairly sure I can't use CloudWatch metrics to approach this.

Is there a way I could accomplish this using Jenkins without creating race conditions and other issues?

Note: My application is not constrained by CPU, Disk I/O, or any other metric provided by CloudWatch; the constraint is the application's ability to process from incoming Kafka topics.

**Tags:** jenkins, amazon-web-services, autoscaling

> **Comments:**
> **username_1**: Maybe you can explain a little more why you think cloudwatch is not a good approach ?  
> **username_0**: My application is not constrained by CPU, Disk I/O, or any other metric provided by CloudWatch; the constraint is the application's ability to process from incoming Kafka topics.  

---

## 1 answers

### Answer by username_2 on 2017-04-04. Score: 7 | Accepted answer

CloudWatch is absolutely [the way to go](https://blog.developer.bazaarvoice.com/2015/04/17/predictively-scaling-ec2-instances-with-custom-cloudwatch-metrics/) on this front, you can surface Kafka Metrics in CloudWatch either by:

1. Having a [separate process](https://github.com/sidcarter/kafka-cloudwatch) that pushes the metrics into CloudWatch.
2. Have your Producers, Consumers or Stream Processors push the metrics you need into CloudWatch.

The process you follow will broadly be broken down into:

1. [Publish Custom Metrics](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/publishingMetrics.html).
2. Create an [AutoScaling Launch Configuration](http://docs.aws.amazon.com/autoscaling/latest/userguide/GettingStartedTutorial.html#gs-create-lc).
3. Create an [AutoScaling Group and Policies](http://docs.aws.amazon.com/autoscaling/latest/userguide/GettingStartedTutorial.html#gs-create-asg) based upon your Custom Metric.

Depending on your cost model I would also consider how you can use spot instances to keep the costs down, i.e. if you have non-critical but high compute workloads use [Spot Instances](https://aws.amazon.com/ec2/spot/) as part of your autoscaling configuration.

> **Comments:**
> **username_3**: The first link is dead  
> **username_0**: @MarquisBlount here's the [Google Cache for the page](https://webcache.googleusercontent.com/search?q=cache:mM9WvsLriCQJ:https://blog.developer.bazaarvoice.com/2015/04/17/predictively-scaling-ec2-instances-with-custom-cloudwatch-metrics/)  

---

