<p>I would personally consider a model like this:</p>

<pre><code>Timed Lambdas -&gt; Checks spot price -&gt; Push to ElastiCache
</code></pre>

<p>Then when you need instances:</p>

<pre><code>Timed lambdas -&gt; Pulls spot price from ElastiCache, sets it as environment variable on your Machine where you spin up IaC from -&gt; This is parsed as argument to IaC code and pushes out the spot price
</code></pre>

<p>You could set some tolerances, too, within the lambdas (i.e. 10, 25, 50% increases based on importance) and a hard cap of on-demand, for example. It's also a great place to build the logic to handle, for example, finding the cheapest AZ, finding the relatively cheapest spot price (<code>2xt2.medium</code> vs <code>t2.large</code>), etc.</p>

CC BY-SA 3.0
2017-04-19T16:16:19.623Z
Just for information, you could have edited the deleted answer and flagged it to ask for undeletion. (No harm done, it's just informative :))
Tensibai
CC BY-SA 3.0
2017-04-19T16:42:05.630Z
I will leave my shame to persevere to remind me of my misguided days for all eternity.
Henry
CC BY-SA 3.0
2017-04-19T19:02:47.697Z
Rofl, there is really no shame, don't worry :)
Tensibai
CC BY-SA 3.0
2017-04-19T19:04:31.557Z
MY SCARS DEFINE WHO I AM
Henry
CC BY-SA 4.0
2018-06-17T03:35:58.587Z
Cool idea - might also be worth mentioning that one could push the price to a DynamoDB table, in case they don't already have an ElastiCache cluster lying around ;). Or maybe even a JSON file in an S3 bucket.
Tim Malone
CC BY-SA 3.0
2017-04-17T18:37:18.980Z
2017-04-17T18:37:18.980Z
Henry
<p>Let me give you a tool-agnostic way of trying to do this.</p>

<blockquote>
  <p>every so often the price spikes and it'll be an hour or two before anyone notices that no new agents are coming up, and then we have to go in manually and check the spot price and adjust accordingly, or switch zones, etc</p>
</blockquote>

<p>We faced the same problem in the infrastructure which we're building. So, we had <code>if-else</code> style blocks to set the bid price, depending on the on-demand price of the instance.</p>

<p>AWS has an <a href="https://aws.amazon.com/blogs/aws/new-aws-price-list-api/" rel="nofollow noreferrer">API</a> for getting the on-demand price of an instance. We used this <a href="https://github.com/erans/ec2instancespricing" rel="nofollow noreferrer">Python wrapper</a> for the purpose.</p>

<p>So, once we got the on-demand price (let's say <code>X</code>), we plugged in <code>if-else</code> style blocks, which are <code>0.4*X</code>, <code>0.6*X</code>, <code>0.8*X</code>, <code>X</code>, which means we are trying for a bid price in the range 40%, 60%, 80% of the on-demand price. If all fails, then we are falling back to creating on-demand instances.</p>

<p>Also, as this is independent of AWS's current spot pricing, we never pay a price above the on-demand price.</p>

<p>But, if you're looking for a way to do it on-the-fly, then Hashfyre's solution should be the way to go.</p>

CC BY-SA 3.0
2017-04-17T17:13:39.677Z
2017-04-17T17:13:39.677Z
Dawny33
<p>Spotted an open source tool called <a href="https://github.com/cristim/autospotting" rel="noreferrer">autospotting</a> that just might help with this:  </p>

<blockquote>
  <p>Once enabled on an existing on-demand AutoScaling group, it launches an EC2 spot instance that is cheaper, at least as large and configured identically to your current on-demand instances. As soon as the new instance is ready, it is added to the group and an on-demand instance is detached from the group and terminated.</p>
</blockquote>

<p>We have this in our to-do pipeline, will be able to add more context once we finish that.</p>

<p>Update:</p>

<p>Another tool that was recently demo'ed in a conference was <a href="https://github.com/mapbox/spotswap" rel="noreferrer">mapbox/spotswap</a></p>

<p>This works slightly differently. It monitors a normal ASG with either On-Demand or Reserved instances and then, if scale arises, bids and provisions spot instances of similar compute level on a separate ASG.</p>

CC BY-SA 3.0
2017-04-17T16:52:33.267Z
Welcome to the site, Hashfyre :)
Dawny33
CC BY-SA 3.0
2017-10-15T22:10:17.330Z
I'm the author of AutoSpotting, thanks for mentioning it! Did you give it a try yet? I'm all ears for user feedback.
Cristian Măgherușan-Stanciu
CC BY-SA 3.0
2017-04-17T16:30:01.173Z
2017-06-08T09:10:25.253Z
2017-06-08T09:10:25.253Z
Hashfyre
<p>I've been thinking about a Jenkins job to check the pricing for the agent boxes we spin up; every so often the price spikes and it'll be an hour or two before anyone notices that no new agents are coming up, and then we have to go in manually and check the spot price and adjust accordingly, or switch zones, etc.</p>

<p>My first instinct was that the best route is to have an hourly Jenkins job that runs a <code>aws ec2 describe-spot-instance-requests</code> and checks for failed requests (and then slacks us on a failure). But I'm wondering if there's a cleaner method that involves actually comparing hard prices (and could therefore tell us exactly what's off, and by how much), rather than looking at successful/failed requests. </p>

<p>Anyone set up something similar? How'd you do it? </p>

CC BY-SA 3.0
2017-04-18T01:23:14.560Z
Reasonable question, I guess... but why would you not just bid what the instances are actually worth to you, select multiple instance types (including instances larger than what you need, which are occasionally cheaper when smaller instance spot prices rise... occasionally -- including right now as I write this -- I find myself with a cc2.8xlarge, because it is currently cheaper than the c3.2xlarge I needed, despite having 4x the cores and 4x the memory) and let the system handle instance type selection and AZ placement based on the market?
Michael - sqlbot
CC BY-SA 3.0
2017-04-18T12:33:43.027Z
@Michael-sqlbot I don't have say in that part of the process; we have documentation stating that we purchase a certain size of instance for auditing purposes and so that's what we're stuck with. Just trying to make the most of an unideal situation.
Alex
CC BY-SA 3.0
2017-04-18T15:44:49.127Z
Fair enough, @Alex.  Just thought I'd throw it in, in case you hadn't considered it.
Michael - sqlbot
CC BY-SA 4.0
2018-06-17T03:34:18.140Z
@Alex Out of interest, is the certain instance size so it can be seen easily on bills? Because you (rather, they!) could use tags and then see them in the Cost Explorer.
Tim Malone
<jenkins><amazon-web-services><amazon-ec2>
How to check AWS spot pricing automatically?
Alex
# How to check AWS spot pricing automatically?

(Asked by: username_0 on 2017-04-17)

I've been thinking about a Jenkins job to check the pricing for the agent boxes we spin up; every so often the price spikes and it'll be an hour or two before anyone notices that no new agents are coming up, and then we have to go in manually and check the spot price and adjust accordingly, or switch zones, etc.

My first instinct was that the best route is to have an hourly Jenkins job that runs a `aws ec2 describe-spot-instance-requests` and checks for failed requests (and then slacks us on a failure). But I'm wondering if there's a cleaner method that involves actually comparing hard prices (and could therefore tell us exactly what's off, and by how much), rather than looking at successful/failed requests.

Anyone set up something similar? How'd you do it?

**Tags:** jenkins, amazon-web-services, amazon-ec2

> **Comments:**
> **username_1**: Reasonable question, I guess... but why would you not just bid what the instances are actually worth to you, select multiple instance types (including instances larger than what you need, which are occasionally cheaper when smaller instance spot prices rise... occasionally -- including right now as I write this -- I find myself with a cc2.8xlarge, because it is currently cheaper than the c3.2xlarge I needed, despite having 4x the cores and 4x the memory) and let the system handle instance type selection and AZ placement based on the market?  
> **username_0**: @Michael-sqlbot I don't have say in that part of the process; we have documentation stating that we purchase a certain size of instance for auditing purposes and so that's what we're stuck with. Just trying to make the most of an unideal situation.  
> **username_1**: Fair enough, @username_0.  Just thought I'd throw it in, in case you hadn't considered it.  
> **username_2**: @username_0 Out of interest, is the certain instance size so it can be seen easily on bills? Because you (rather, they!) could use tags and then see them in the Cost Explorer.  

---

## 3 answers

### Answer by username_6 on 2017-04-17. Score: 15

Spotted an open source tool called [autospotting](https://github.com/cristim/autospotting) that just might help with this:

> Once enabled on an existing on-demand AutoScaling group, it launches an EC2 spot instance that is cheaper, at least as large and configured identically to your current on-demand instances. As soon as the new instance is ready, it is added to the group and an on-demand instance is detached from the group and terminated.

We have this in our to-do pipeline, will be able to add more context once we finish that.

Update:

Another tool that was recently demo'ed in a conference was [mapbox/spotswap](https://github.com/mapbox/spotswap)

This works slightly differently. It monitors a normal ASG with either On-Demand or Reserved instances and then, if scale arises, bids and provisions spot instances of similar compute level on a separate ASG.

> **Comments:**
> **username_5**: Welcome to the site, username_6 :)  
> **username_7**: I'm the author of AutoSpotting, thanks for mentioning it! Did you give it a try yet? I'm all ears for user feedback.  

---

### Answer by username_3 on 2017-04-17. Score: 6

I would personally consider a model like this:
Timed Lambdas -> Checks spot price -> Push to ElastiCache

Then when you need instances:
Timed lambdas -> Pulls spot price from ElastiCache, sets it as environment variable on your Machine where you spin up IaC from -> This is parsed as argument to IaC code and pushes out the spot price

You could set some tolerances, too, within the lambdas (i.e. 10, 25, 50% increases based on importance) and a hard cap of on-demand, for example. It's also a great place to build the logic to handle, for example, finding the cheapest AZ, finding the relatively cheapest spot price (2xt2.medium vs t2.large), etc.

> **Comments:**
> **username_4**: Just for information, you could have edited the deleted answer and flagged it to ask for undeletion. (No harm done, it's just informative :))  
> **username_3**: I will leave my shame to persevere to remind me of my misguided days for all eternity.  
> **username_4**: Rofl, there is really no shame, don't worry :)  
> **username_3**: MY SCARS DEFINE WHO I AM  
> **username_2**: Cool idea - might also be worth mentioning that one could push the price to a DynamoDB table, in case they don't already have an ElastiCache cluster lying around ;). Or maybe even a JSON file in an S3 bucket.  

---

### Answer by username_5 on 2017-04-17. Score: 4

Let me give you a tool-agnostic way of trying to do this.

> every so often the price spikes and it'll be an hour or two before anyone notices that no new agents are coming up, and then we have to go in manually and check the spot price and adjust accordingly, or switch zones, etc

We faced the same problem in the infrastructure which we're building. So, we had `if-else` style blocks to set the bid price, depending on the on-demand price of the instance.

AWS has an [API](https://aws.amazon.com/blogs/aws/new-aws-price-list-api/) for getting the on-demand price of an instance. We used this [Python wrapper](https://github.com/erans/ec2instancespricing) for the purpose.

So, once we got the on-demand price (let's say `X`), we plugged in `if-else` style blocks, which are `0.4*X`, `0.6*X`, `0.8*X`, `X`, which means we are trying for a bid price in the range 40%, 60%, 80% of the on-demand price. If all fails, then we are falling back to creating on-demand instances.

Also, as this is independent of AWS's current spot pricing, we never pay a price above the on-demand price.

But, if you're looking for a way to do it on-the-fly, then username_6's solution should be the way to go.


---

