<p>If your environments are per customer, I would suggest <em>in your specific case</em> to have a <strong>repository per customer</strong>. (In general it is repository per environment.) This repository would have a standard directory structure for environment variables, ansible variables and inventories, strongly encrypted secrets (account access tokens, private keys, etc.).  You would git submodule the code into those repositories. I would probably do it in multiple repositories. One for ansible roles and modules, one for maintenance and deployment scripts, one for each major application running in the environments.</p>

<p>Now you can optionally actually fork the code or otherwise pin the submodule at specific tag for <strong>release</strong>, making sure that the code managing the customer's environment would not change unless tested and released.</p>

<p>If you are using an <strong>artifact repository</strong>, make sure the artifacts are properly versioned and those versions are specified in the environment variables properly.</p>

<p><strong>Automation</strong> is important because the environment variables should not be updated by humans where possible, but generated by scripts. Make sure that there are nearly no manual updates in the per customer inventory and developers only update the code repositories. If they want to make configuration change, it should be done to one of the generating scripts, which is then run to generate the variables and the diff is committed into the customer repository. It pays to setup continuous integration for this process. Without this at some point there will be too many repositories to <strong>maintain</strong>.</p>

CC BY-SA 3.0
2017-04-06T05:22:24.760Z
Only one objection: secrets shouldn't go into a version control repository unless it has strict access control support. Git doesn't - whoever pulls the repository can see the secrets, which can be a problem - they're no longer secrets.
Dan Cornilescu
CC BY-SA 3.0
2017-04-06T11:57:11.313Z
Good catch. It is encrypted secrets. The decryption keys are tramsient.
Jiri Klouda
CC BY-SA 3.0
2017-04-06T05:13:46.063Z
2017-04-06T11:58:39.237Z
2017-04-06T11:58:39.237Z
Jiri Klouda
<p>I've had two runs at doing environment variables in a scalable way and neither has ended up perfect because,  as I've discovered, is a very tricky thing to get right. I'll give a summary of both of my experiences below:</p>

<p><strong>Common Factors</strong></p>

<ul>
<li>Environment variables are stored in a separate repository from the original source code (they are submoduled together but are still based on separate repos)</li>
<li>There is a separate "build" process for the artifact and it's variables.</li>
<li>There is <strong>not</strong> a separate release process for the environment variables. If you want to change environment variables you need to go through the same change review boards and usual</li>
</ul>

<p><strong>Using Consul KV Pairs</strong> </p>

<p>The environment variables are loaded from an artifact repository (never the original git repo) and loaded into a namespaced KV pair tree, for example</p>

<p>/env/dev1/my/application/v1.1.1</p>

<p>Where the preceding dev1 is the name of the environment, the my/application is the application namespace, and the v1.1.1 is the version of the environment variables to use.</p>

<p>To developers all these things are invisible. At runtime the platform checks that the environment exists in the current consul cluster (if it doesn't there's a problem and it's errors out), it then checks the subtree for the applications namespace (this way there can be no cross contamination where one app references another apps vars) then the version number of the configuration is taken from label connected to the deployable artifact. Updating this label is the key thing here because it means if we lost both production data centers we could stand up the environment again by simple reading the meta data from our deployable artifacts and loading <em>all the environment variables</em> into the KV store.</p>

<p><em>Problems With This Approach</em>
Developers always, and I mean every single time, found a way to slip configuration changes into the environment that had significant impacts to how the application would run. Because it always ended up being easier to get configuration changes approved than code changes.</p>

<p><strong>Storing a "Deployment" Artifact with variables embedded</strong></p>

<p>This tightly couples the exact version of the artifact to the version of configuration. If you changed configuration then you had to rebuild this deployment artifact.</p>

<p>The deployment artifact itself was essentially a yaml file which contained the URL to the releasable binary and all the configuration attached to it.</p>

<p>The platform contains componentry to read variables and then out them into the process tree of the application when it starts up.</p>

<p>This has so far been a lot more successful because there is an artifact we can trace the history of and which we can hold up to any review board and say "this is the only artifact we care about, we don't need t look at any other changes, only changes to this thing" (ie. Version of application to deploy,  environment variables included etc.</p>

<p>This makes it just a little bit harder for developers to try and build logic into their application that will change its behavior based on variables so they can slip in changes without going through appropriate testing cycles.</p>

<p><strong>Bonus Points</strong></p>

<p>Consider application secrets. Our solution to this has so far been to provide a public RSA key which development teams use to encrypt an extended Java key store (almost every language has a library somewhere that can read Java keys stores) this is then treated like a third type of artifact and is pulled onto the server, decrypted with our platform private key, and provided to the application at run time.</p>

<p>Admittedly secrets management is its own can of worms. But it's probably worth considering.</p>

CC BY-SA 3.0
2017-03-14T22:04:25.083Z
Re: application secrets, I would suggest taking a look at Vault (https://www.vaultproject.io/) as it is also a part of Hashicorp's toolchain and integrates rather neatly with Consul (and other tools from that box)
Michael Bravo
CC BY-SA 3.0
2017-03-17T16:05:28.247Z
I've actually been very underwhelmed by vault given how great hashicorp stuff usually is. Essentially three major gaps in their product vs the rest of the market - 1. 'Secrets for secrets' is essentially what the model boils down to. I get sharding or using an HSM. But essentially it's just secrets trading. 2. Tool compatibility, unlike their other tools there's no support for plugins 3. Price. I didn't believe it when told the company I'm with thought vault was expensive. They've rejected products for being too cheap, it's messed up. But vault was so much that they didn't even consider it.
hvindin
CC BY-SA 3.0
2017-03-22T20:13:43.510Z
It's worth noting that it's only cost-prohibitive *if you use a paid version*. Vault's core product is open-source. Of course they don't list pricing for pro/enterprise versions on their site, so I have no idea how [un]reasonable it may be for those editions.
Adrian
CC BY-SA 3.0
2017-03-22T20:54:29.737Z
Hm, I didn't notice that omission from my comment, although to be fair my first two problems with vault still stand. Although, to qualify, those are my thoughts on vault compared to other hashicorp products, all of which I think are pretty great. Compared to other products on the market the perform a similar function is probably on par, just for some reason a lot more expensive than expected.
hvindin
CC BY-SA 3.0
2017-03-23T16:42:40.337Z
Can you give an example of "build logic into their application that will change its behavior based on variables so they can slip in changes without going through appropriate testing cycles."? Sound like something really common, but I just can't imagine a concrete example.
kenchew
CC BY-SA 3.0
2017-03-25T12:18:01.417Z
A simple example: `if (env == "prod" ) { prod things; } else { non prod things; }`. No chance a good test case gets written for that. I don't have any examples on me and nothing springs to mind, I'll see if I can find some examples when I'm at work which may be helpful. Where it starts to get nasty is with a recent trend of using ruby for config, allowing procedures to be stored as config values. My general rule is: devs change code, admins change config, business users update a database. Even if Devops blurs the dev/admin line, you should know if you are administering an app or developing it.
hvindin
CC BY-SA 3.0
2017-03-12T21:35:15.127Z
2017-03-20T15:48:38.973Z
2017-03-20T15:48:38.973Z
hvindin
<p>This is a very broad question about methods and advice regarding environment variables / structure. But ultimately I'm looking for answers for the very specific question of 'How should I store my environment variables?'</p>

<p>Firstly some clarifications:</p>

<ul>
<li>An environment to me can be from 3 to 10 servers and is a way of containing a specific customer's infrastructure.</li>
<li>Inside each environment there are a few variables which are mostly automatically generated from a few key inputs (name, size etc).</li>
</ul>

<p>As it stands right now we're storing all of our environment variables in a structure like so:</p>

<pre><code>&lt;playbook&gt;.yml                   # Various playbooks for deployment
roles/windows                    # Ansible role for Ubuntu
roles/ubuntu                     # Ansible role for Ubuntu
config/hosts/&lt;name&gt;.yml          # Ansible inventory
config/hosts/vars/&lt;name&gt;.json    # Environment specific variables 
</code></pre>

<p>Right now the config is initialised as a submodule in the git repository above. As the variables file changes rather frequently this has caused issues with the data changing, once, twice or even three times between commits making changes increasingly difficult to trace.</p>

<p>As I personally see it going forward we should be looking to store all our customer variables in a centralised / scalable way and then hook into it with a dynamic inventory with <a href="/questions/tagged/ansible" class="post-tag" title="show questions tagged &#39;ansible&#39;" rel="tag">ansible</a>.</p>

<p>I understand there are a few technologies which seem to do a part of what could be required such as Consul but they seem to work best in an environment which serves one big application rather than lots of smaller slightly differing ones.</p>

<p>I essentially see us having to write an inventory script and then just shove all of our data into some not-for-purpose built database and then keep going on as if nothing changed. I see this conceivably as a way to potentially trim down on a lot of the data we currently store and perhaps look into different ways of storing the data rather than just scaling up what serves it again.</p>

<p>I'm hoping somebody has some kind of experience in implementing infrastructure as code when having to deal with lots of smaller environments as opposed to one, two or three huge ones.</p>

<p>Any suggestions?</p>

<ansible><configuration><databases><provisioning><environment-variables>
How should I store my environment variables?
Naphta
# How should I store my environment variables?

(Asked by: username_0 on 2017-03-11)

This is a very broad question about methods and advice regarding environment variables / structure. But ultimately I'm looking for answers for the very specific question of 'How should I store my environment variables?'
Firstly some clarifications:

An environment to me can be from 3 to 10 servers and is a way of containing a specific customer's infrastructure.
Inside each environment there are a few variables which are mostly automatically generated from a few key inputs (name, size etc).

As it stands right now we're storing all of our environment variables in a structure like so:
.yml                   # Various playbooks for deployment
roles/windows                    # Ansible role for Ubuntu
roles/ubuntu                     # Ansible role for Ubuntu
config/hosts/.yml          # Ansible inventory
config/hosts/vars/.json    # Environment specific variables 

Right now the config is initialised as a submodule in the git repository above. As the variables file changes rather frequently this has caused issues with the data changing, once, twice or even three times between commits making changes increasingly difficult to trace.
As I personally see it going forward we should be looking to store all our customer variables in a centralised / scalable way and then hook into it with a dynamic inventory with ansible.
I understand there are a few technologies which seem to do a part of what could be required such as Consul but they seem to work best in an environment which serves one big application rather than lots of smaller slightly differing ones.
I essentially see us having to write an inventory script and then just shove all of our data into some not-for-purpose built database and then keep going on as if nothing changed. I see this conceivably as a way to potentially trim down on a lot of the data we currently store and perhaps look into different ways of storing the data rather than just scaling up what serves it again.
I'm hoping somebody has some kind of experience in implementing infrastructure as code when having to deal with lots of smaller environments as opposed to one, two or three huge ones.
Any suggestions?

**Tags:** ansible, configuration, databases, provisioning, environment-variables


---

## 2 answers

### Answer by username_3 on 2017-03-12. Score: 13

I've had two runs at doing environment variables in a scalable way and neither has ended up perfect because, as I've discovered, is a very tricky thing to get right. I'll give a summary of both of my experiences below:

**Common Factors**

* Environment variables are stored in a separate repository from the original source code (they are submoduled together but are still based on separate repos)
* There is a separate "build" process for the artifact and it's variables.
* There is **not** a separate release process for the environment variables. If you want to change environment variables you need to go through the same change review boards and usual

**Using Consul KV Pairs**

The environment variables are loaded from an artifact repository (never the original git repo) and loaded into a namespaced KV pair tree, for example

/env/dev1/my/application/v1.1.1

Where the preceding dev1 is the name of the environment, the my/application is the application namespace, and the v1.1.1 is the version of the environment variables to use.

To developers all these things are invisible. At runtime the platform checks that the environment exists in the current consul cluster (if it doesn't there's a problem and it's errors out), it then checks the subtree for the applications namespace (this way there can be no cross contamination where one app references another apps vars) then the version number of the configuration is taken from label connected to the deployable artifact. Updating this label is the key thing here because it means if we lost both production data centers we could stand up the environment again by simple reading the meta data from our deployable artifacts and loading *all the environment variables* into the KV store.

*Problems With This Approach*
Developers always, and I mean every single time, found a way to slip configuration changes into the environment that had significant impacts to how the application would run. Because it always ended up being easier to get configuration changes approved than code changes.

**Storing a "Deployment" Artifact with variables embedded**

This tightly couples the exact version of the artifact to the version of configuration. If you changed configuration then you had to rebuild this deployment artifact.

The deployment artifact itself was essentially a yaml file which contained the URL to the releasable binary and all the configuration attached to it.

The platform contains componentry to read variables and then out them into the process tree of the application when it starts up.

This has so far been a lot more successful because there is an artifact we can trace the history of and which we can hold up to any review board and say "this is the only artifact we care about, we don't need t look at any other changes, only changes to this thing" (ie. Version of application to deploy, environment variables included etc.

This makes it just a little bit harder for developers to try and build logic into their application that will change its behavior based on variables so they can slip in changes without going through appropriate testing cycles.

**Bonus Points**

Consider application secrets. Our solution to this has so far been to provide a public RSA key which development teams use to encrypt an extended Java key store (almost every language has a library somewhere that can read Java keys stores) this is then treated like a third type of artifact and is pulled onto the server, decrypted with our platform private key, and provided to the application at run time.

Admittedly secrets management is its own can of worms. But it's probably worth considering.

> **Comments:**
> **username_4**: Re: application secrets, I would suggest taking a look at Vault (https://www.vaultproject.io/) as it is also a part of Hashicorp's toolchain and integrates rather neatly with Consul (and other tools from that box)  
> **username_3**: I've actually been very underwhelmed by vault given how great hashicorp stuff usually is. Essentially three major gaps in their product vs the rest of the market - 1. 'Secrets for secrets' is essentially what the model boils down to. I get sharding or using an HSM. But essentially it's just secrets trading. 2. Tool compatibility, unlike their other tools there's no support for plugins 3. Price. I didn't believe it when told the company I'm with thought vault was expensive. They've rejected products for being too cheap, it's messed up. But vault was so much that they didn't even consider it.  
> **username_5**: It's worth noting that it's only cost-prohibitive *if you use a paid version*. Vault's core product is open-source. Of course they don't list pricing for pro/enterprise versions on their site, so I have no idea how [un]reasonable it may be for those editions.  
> **username_3**: Hm, I didn't notice that omission from my comment, although to be fair my first two problems with vault still stand. Although, to qualify, those are my thoughts on vault compared to other hashicorp products, all of which I think are pretty great. Compared to other products on the market the perform a similar function is probably on par, just for some reason a lot more expensive than expected.  
> **username_6**: Can you give an example of "build logic into their application that will change its behavior based on variables so they can slip in changes without going through appropriate testing cycles."? Sound like something really common, but I just can't imagine a concrete example.  
> **username_3**: A simple example: `if (env == "prod" ) { prod things; } else { non prod things; }`. No chance a good test case gets written for that. I don't have any examples on me and nothing springs to mind, I'll see if I can find some examples when I'm at work which may be helpful. Where it starts to get nasty is with a recent trend of using ruby for config, allowing procedures to be stored as config values. My general rule is: devs change code, admins change config, business users update a database. Even if Devops blurs the dev/admin line, you should know if you are administering an app or developing it.  

---

### Answer by username_1 on 2017-04-06. Score: 3

If your environments are per customer, I would suggest *in your specific case* to have a **repository per customer**. (In general it is repository per environment.) This repository would have a standard directory structure for environment variables, ansible variables and inventories, strongly encrypted secrets (account access tokens, private keys, etc.). You would git submodule the code into those repositories. I would probably do it in multiple repositories. One for ansible roles and modules, one for maintenance and deployment scripts, one for each major application running in the environments.

Now you can optionally actually fork the code or otherwise pin the submodule at specific tag for **release**, making sure that the code managing the customer's environment would not change unless tested and released.

If you are using an **artifact repository**, make sure the artifacts are properly versioned and those versions are specified in the environment variables properly.

**Automation** is important because the environment variables should not be updated by humans where possible, but generated by scripts. Make sure that there are nearly no manual updates in the per customer inventory and developers only update the code repositories. If they want to make configuration change, it should be done to one of the generating scripts, which is then run to generate the variables and the diff is committed into the customer repository. It pays to setup continuous integration for this process. Without this at some point there will be too many repositories to **maintain**.

> **Comments:**
> **username_2**: Only one objection: secrets shouldn't go into a version control repository unless it has strict access control support. Git doesn't - whoever pulls the repository can see the secrets, which can be a problem - they're no longer secrets.  
> **username_1**: Good catch. It is encrypted secrets. The decryption keys are tramsient.  

---

