<p>When you're running ephemeral workloads like you would on a spot instance the best bet, if you don't want to bake an ami, is going to be a secret store of some kind.</p>

<p>The two most common methods I've seen for doing this are:</p>

<ol>
<li><p>If your provisioning tool (<a href="https://www.terraform.io/docs/providers/aws/r/kms_key.html" rel="nofollow noreferrer">terraform</a>, <a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-kms-key.html" rel="nofollow noreferrer">cloudformation</a>, <a href="https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.boto_kms.html" rel="nofollow noreferrer">salt cloud</a>) allows it then you can create a kms key and use it to encrypt secrets in user data. Once the instance has started it can decrypt them using that key.</p></li>
<li><p>You can use a secret store like you mentioned. You said that AWS Secrets Manager would get pricey so you're probably going to want to use <a href="https://www.vaultproject.io/" rel="nofollow noreferrer">Vault</a>. It's not <em>that</em> hard to set up and can use IAM permissions to allow your instances access to secrets. Using it also opens other avenues to doing secret storage right.</p></li>
</ol>

CC BY-SA 4.0
2019-02-22T14:22:03.560Z
2019-02-22T14:22:03.560Z
Briansbum
<p>Right now, we only use EC2 on-demand instances, and we provision sensitive data (credentials, private keys, etc.) on the instances' EBS which are encrypted with a KMS key.</p>

<p>My understanding is that I have a few ways to pass those secret data to Spot instances:</p>

<ul>
<li>Burn them in an encrypted AMI (but then, what about host-specific stuff, like SSH server private keys?)</li>
<li>Pass them as UserData when we call RunInstances (but then we would need to encrypt those data, and decrypt them with a secret burnt in the AMI)</li>
<li>Use Secrets Manager, but that could quickly cost a lot...</li>
<li>Use Parameter Store, and encrypt the payload with a KMS key (1 shared key for shared data + 1 key per node for node specific data?)</li>
</ul>

<p>And I am not sure I could individually identify spot instances?</p>

<p>Am I missing something there? How do you solve those challenges?</p>

CC BY-SA 4.0
2019-02-22T06:26:25.463Z
A variant of instance Ebs is to share an EFS file system
jdog
CC BY-SA 4.0
2019-02-22T08:03:21.900Z
*"lot of engineering there"* Are you sure? https://docs.aws.amazon.com/kms/latest/developerguide/services-parameter-store.html
Michael - sqlbot
CC BY-SA 4.0
2019-02-22T11:51:33.243Z
@Michael-sqlbot Thanks for your feedback, I rephrased.
Philippe
CC BY-SA 4.0
2019-02-22T11:52:24.733Z
But maybe I am just wrong to think about node-specific data and I should accept that all nodes are the _same_? It's very hard for me as I cannot, for example, allow myself to blindly trust an unknown SSH host key.
Philippe
CC BY-SA 4.0
2019-02-22T14:02:32.283Z
A quick one with the ssh keys, you don't need to trust unknown ssh keys. I've used the approach linked below a lot and it works _really_ well, it's simple to implement and secure enough. Also it's worlds easier to maintain than a key per instance or user.
https://code.fb.com/security/scalable-and-secure-access-with-ssh/
Briansbum
CC BY-SA 4.0
2019-02-22T15:11:29.650Z
@Philippe I see what you mean, now.  Thank you for the clarification.  But it does feel like you might 
be overcomplicating the issue.  SSM Parameter Store allows granular access control of *which* parameters (by path prefix) any given set of credentials can access.  If your instances are doing the same work, there should be no need for them to have individual "personalities" -- they can share one IAM role that allows them to access the relevant parameters, by path, and SSMPS/KMS integration can use the shared KMS key.  You can't decrypt what you can't access.
Michael - sqlbot
<amazon-ec2><security><devsecops>
How to securely pass sensitive data to EC2 Spot instances?
Philippe
# How to securely pass sensitive data to EC2 Spot instances?

(Asked by: username_0 on 2019-02-22)

Right now, we only use EC2 on-demand instances, and we provision sensitive data (credentials, private keys, etc.) on the instances' EBS which are encrypted with a KMS key.

My understanding is that I have a few ways to pass those secret data to Spot instances:

* Burn them in an encrypted AMI (but then, what about host-specific stuff, like SSH server private keys?)
* Pass them as UserData when we call RunInstances (but then we would need to encrypt those data, and decrypt them with a secret burnt in the AMI)
* Use Secrets Manager, but that could quickly cost a lot...
* Use Parameter Store, and encrypt the payload with a KMS key (1 shared key for shared data + 1 key per node for node specific data?)

And I am not sure I could individually identify spot instances?

Am I missing something there? How do you solve those challenges?

**Tags:** amazon-ec2, security, devsecops

> **Comments:**
> **username_1**: A variant of instance Ebs is to share an EFS file system  
> **username_2**: *"lot of engineering there"* Are you sure? https://docs.aws.amazon.com/kms/latest/developerguide/services-parameter-store.html  
> **username_0**: @Michael-sqlbot Thanks for your feedback, I rephrased.  
> **username_0**: But maybe I am just wrong to think about node-specific data and I should accept that all nodes are the _same_? It's very hard for me as I cannot, for example, allow myself to blindly trust an unknown SSH host key.  
> **username_3**: A quick one with the ssh keys, you don't need to trust unknown ssh keys. I've used the approach linked below a lot and it works _really_ well, it's simple to implement and secure enough. Also it's worlds easier to maintain than a key per instance or user. https://code.fb.com/security/scalable-and-secure-access-with-ssh/  
> **username_2**: @username_0 I see what you mean, now.  Thank you for the clarification.  But it does feel like you might  be overcomplicating the issue.  SSM Parameter Store allows granular access control of *which* parameters (by path prefix) any given set of credentials can access.  If your instances are doing the same work, there should be no need for them to have individual "personalities" -- they can share one IAM role that allows them to access the relevant parameters, by path, and SSMPS/KMS integration can use the shared KMS key.  You can't decrypt what you can't access.  

---

## 1 answers

### Answer by username_3 on 2019-02-22. Score: 3 | Accepted answer

When you're running ephemeral workloads like you would on a spot instance the best bet, if you don't want to bake an ami, is going to be a secret store of some kind.

The two most common methods I've seen for doing this are:

1. If your provisioning tool ([terraform](https://www.terraform.io/docs/providers/aws/r/kms_key.html), [cloudformation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-kms-key.html), [salt cloud](https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.boto_kms.html)) allows it then you can create a kms key and use it to encrypt secrets in user data. Once the instance has started it can decrypt them using that key.
2. You can use a secret store like you mentioned. You said that AWS Secrets Manager would get pricey so you're probably going to want to use [Vault](https://www.vaultproject.io/). It's not *that* hard to set up and can use IAM permissions to allow your instances access to secrets. Using it also opens other avenues to doing secret storage right.


---

