<p>I am a big fan of saltstack, mainly because it provides a comprehensive solution regarding the whole server life cycle, unlike ansible which need another tool (such as terraform) to provision vms.</p>

<p>salt + salt-cloud is one very good suite from my opinion.</p>

<p>I do not see any differences regarding the amount of work required by either tool (chef, puppet, ansible, salt) to get a working IAC system in place, this is, of course, according to your language preference.
If you're in python then go for salt or ansible, otherwise go check puppet or chef.
Do not forget to plug your iac code base to a VCS.</p>

CC BY-SA 4.0
2019-11-06T22:16:29.743Z
The push (salt/ansible) vs pull (chef/puppet) model difference may play a role in the choice too (for DMZ systems for example)
Tensibai
CC BY-SA 4.0
2019-11-06T22:23:55.000Z
salt has an interesting ability for both model,  pull (salt-call executed from the minion) and push (salt command executed on the master), this helps a lot for minion with an unstable connection.
Pier
CC BY-SA 4.0
2019-11-06T22:34:19.383Z
Missed that model in salt, I didn't check it for a while that said, it still use the system python (as ansible) if I get it right, which make it quite complicated when running legacy apps incompatible with its requirements (Chef has a point here coming with its own ruby independent of the system one at the cost of a larger client of course)
Tensibai
CC BY-SA 4.0
2019-11-06T14:59:27.703Z
2019-11-06T14:59:27.703Z
Pier
<p>This question runs two risks:</p>

<ol>
<li>of being marked as "primarily opinion-based", and</li>
<li>its answers might age poorly.</li>
</ol>

<p>However, I think it's useful to express an opinion, if only for posterity; if and when things change, there may be better answers -- but there will always be blank slates.</p>

<p>I would split the problem up into four sections:</p>

<ol>
<li>Infrastructure</li>
<li>Application Building, testing and integration</li>
<li>Deployment</li>
<li>Monitoring, logging and analytics</li>
</ol>

<p>Furthermore, let's make this cloud-independent (so, this toolkit should be valid for any or all of AWS, GCE, Azure <em>etc</em>, as well as on-prem OpenStack, bare metal cloud (e.g. <a href="https://www.packet.com/" rel="noreferrer">Packet</a>, etc).</p>

<h2>Infrastructure</h2>

<p>Infrastructure components consist of </p>

<ol>
<li><p>Your environments (pre, prod, integration, testing, etc - however you define them). These should be built with Terraform modules.</p>

<ol>
<li>Each module in its own repo</li>
<li>Consider carefully separation of state, so that you can easily create and destroy components and keep costs down</li>
</ol></li>
<li><p>"DevOps" infrastructure. This includes:</p>

<ol>
<li>code repos</li>
<li>CI/CD instances (controller, agent pools, <em>etc</em>) - <em>e.g.</em> Jenkins master with cloud agent pools.</li>
<li>Deployment runners (see later). </li>
<li>Monitoring, logging and analytics.
Again, build these with terraform modules. As a example, have a Terraform module for each type of application or component.</li>
</ol></li>
<li><p>Team, messaging and support infrastructure. This is often overlooked, but choosing the right tool for the job on Day 2 is hard. It's better to communicate to the stakeholders how issues and change requests should be made. I would venture a combination of GitHub issues and Slack at the very least.</p></li>
<li>secrets management. You will need to pass around a lot of secrets, if you want to foster collaboration. In order to do this securely, you will need a place to manage them and delegate access to and usage of them.</li>
</ol>

<h2>Building, testing, integration.</h2>

<p>These happen in Jenkins. Each application should contain a subdirectory <code>.pipeline</code> with the parts necessary to build it. The part which concerns us here is everything before prod.</p>

<p>I would suggest a pipeline as code approach with a <code>Jenkinsfile</code> in the repo declaring the steps for</p>

<ol>
<li>building the application:</li>
<li>Testing the application

<ol>
<li>Unit tests</li>
<li>Functional tests</li>
<li>Integration tests</li>
<li>(Regression tests)</li>
</ol></li>
<li>Building artifacts

<ol>
<li>building and signing artifacts</li>
<li>Shipping them to the artifactory, with new version info and other metadata</li>
</ol></li>
<li>Building instance images (if your application is "installed" in an environment, and not serverless).</li>
<li>Enforcing changes to the Infrastructure (terraform plan/apply)</li>
<li>Enforcing changes to the deployment configuration (capacity, <em>etc</em>)</li>
</ol>

<p>At the end of this stage, you'll have a properly configured environment (say, pre or prod), an artifact, and an image which you can deploy. If need be, you can run performance tests (with Taurus) or acceptance tests (with cucumber) on them with a job in Jenkins.</p>

<h2>Deployment</h2>

<p>I would vote for a hard separation between the previous phase and this one. Deployment should be done with a different tool and the right tool for the job is in my opinion Spinnaker. The configuration for the deployment should be kept in the same repo as the application (so that developers can serve themselves). Deployments should be done on every change to the codebase, declaratively -- Spinnaker has a JSON syntax for this, making it easy to check consistency and correctness of the deployment scenario.</p>

<h2>Monitoring, logging and analytics</h2>

<p>As mentioned before, Day 1 is a blank slate, but it is quickly dirtied. The DevOps doesn't stop on Day 2 (ie, when apps are deployed), and it would be wise to plan ahead.</p>

<p>The SRE book gives <a href="https://landing.google.com/sre/sre-book/chapters/service-level-objectives/" rel="noreferrer">good advice for instrumenting and monitoring the applications</a> you will be deploying, but this space is very crowded with services and tools claiming solve your monitoring and alerting needs. Again going with the principle that we should be able to deploy anywhere, I would go with a Prometheus for indicator and metric extraction and Grafana for displaying setup. However, you should definitely consider this part wisely.</p>

<hr>

<p>So - the toolkit looks like:</p>

<ol>
<li><a href="https://www.vaultproject.io/" rel="noreferrer">Vault</a> for secrets</li>
<li><a href="https://github.com/" rel="noreferrer">GitHub</a> for code, projects, tasks and issues</li>
<li><a href="https://slack.com" rel="noreferrer">Slack</a> for communication</li>
<li><a href="https://terraform.io" rel="noreferrer">Terraform</a> for infrastructure</li>
<li><a href="https://www.ansible.com" rel="noreferrer">Ansible</a> for configuration management</li>
<li><a href="https://packer.io" rel="noreferrer">Packer</a> for image baking</li>
<li><a href="https://quay.io" rel="noreferrer">Quay</a> for (docker) image registry</li>
<li><a href="https://inspec.io" rel="noreferrer">Inspec</a> for compliance and correctness of state</li>
<li><a href="https://cucumber.io/" rel="noreferrer">Cucumber</a> for Acceptance Testing</li>
<li><a href="https://jenkins.io" rel="noreferrer">Jenkins</a> for pipeline execution</li>
<li><a href="https://www.spinnaker.io/" rel="noreferrer">Spinnaker</a> for Deployment</li>
<li><a href="https://prometheus.io" rel="noreferrer">Prometheus</a> for metrics <a href="https://grafana.com/" rel="noreferrer">Grafana</a> for visualisation and dashboards</li>
<li><a href="http://gettaurus.org/" rel="noreferrer">Taurus</a> (Blazetool) for performance testing</li>
</ol>

<p>Is this a lot of tools? Yes.</p>

<p>Will you be able to deploy anywhere? Also yes.</p>

<p>Will you need <strong>all</strong> of them? probably not. Also, I've chosen tools which are mostly free to use and open source.</p>

<p><strong>Also note:</strong> in this scenario, the entire state of all applications can and should be represented as code, and can thus have semantic versioning (or other kind of versioning which makes sense to you). Any changes to that state can be represented by changes to the code, and can thus have good source code management principles applied to it, such as code review, automated testing, <em>etc</em>.</p>

CC BY-SA 4.0
2019-11-06T15:16:13.227Z
Thank you for the highly detailed answer. I think the packet tool sounds quite promising. The main brain picker really is how development is going to stay consistant as possible between projects and in line with UAT. The plan as mentioned is to keep UAT and PROD the same and only change with deployments to ensure no issues can come about. I can see myself using packer to deploy set images based on requirements so if issues were to come about that could help replicate a bug along side the dev instances if they are developing on the same packet image they are pulling from if that makes sense?
Nabil Aziz
CC BY-SA 4.0
2019-11-06T15:18:24.237Z
We follow the pattern of re-using the Ansible roles, and setting environment-specific variables, to change configuration. The playbooks can then largely be re-used and the Ansible roles independently tested. In packer, we typically build all environments at the same time, so that they are linked back to a git commit.
Bruce Becker
CC BY-SA 4.0
2019-11-06T15:23:25.377Z
Not sure i understand your wording here, are you saying you use ansible to build the consistent enviorments and packet is used to build them at the same time?
Nabil Aziz
CC BY-SA 4.0
2019-11-06T15:25:38.407Z
(note : Packer is a tool from Hashicorp to build images. Packet.net is a bare-metal cloud provider -- these are different :) ). I'm afraid the mods will shut us down here for using the comments to chat, so perhaps it would be best to ask a different question to clarify this.
Bruce Becker
CC BY-SA 4.0
2019-11-06T15:42:14.573Z
Its okay i understand it now, so as a high level overview i could use terraform to provision the servers with packer which basically installs things like nginx and stuff before the server is actually deployed then can use ansible actually setup the specifics.
Nabil Aziz
CC BY-SA 4.0
2019-11-06T14:55:03.953Z
2019-11-06T15:19:16.217Z
2019-11-06T15:19:16.217Z
Bruce Becker
<p>so I have 8 months of DevOps experience previously working as a desktop engineer for two and a half years.Starting a new role in the education sector, recently they brought their IT department which previously worked in different projects/faculties and now they have all been brought together.So this is where I come in to help dictate the application development process by designing and maintaining the infrastructure to build an efficient route to live regardless of the project. So this is essentially a blank slate.</p>

<p>As a very high level overview</p>

<p>So my ideal setup would be creating a production environment for whatever project, replicating the stack and using that as UAT. This would be done through Terraform and Ansible. The Dev estate I am not too sure how this is going to be setup yet but I was thinking about learning docker and k8s to bring up a dev estate.</p>

<p>The way I would look into setting up a CI/CD pipeline using Jenknnis and actioned through Azure DevOps as this is what they use, I would ensure code pushed from the DEV branch is deployed to UAT and tested then deployed to replicate production servers and once tested DNS is switched over to the new servers and the old ones are brought down.</p>

<p>The way how the code is deployed is through building the codebase on the same server Jenkins would be setup on, sending a tar'd file of the code to AWS CodeDeploy to be taken to the next environment. </p>

<p>Through caveats include no changes or access will be allowed on production or UAT servers, any requested changes will be done through terraform and ansible to keep things consistent.</p>

<p>Personally with my limited experience I have used Terraform and ansible and I have felt this provides alot of the work to have that infrastructure as code system in place. At the same time I understand there are tools like chef, puppet and salt which would probably make the job easier but that is where my question is what would you guys want and how would you want to implement it. I should finally mention we seem to be using Azure DevOps as the main tool to setup and manage the pipelines.</p>

CC BY-SA 4.0
2019-11-07T08:38:01.313Z
Could you make this question more generic? Please process the feedback as provided in one of the answers by BruceBecker. Once done, please leave a comment so this question can be evaluated again.
030
<amazon-web-services><ansible><terraform><azure-devops>
Given a blank slate, tool recommendations?
Nabil Aziz
# Given a blank slate, tool recommendations?

(Asked by: username_0 on 2019-11-05)

so I have 8 months of DevOps experience previously working as a desktop engineer for two and a half years.Starting a new role in the education sector, recently they brought their IT department which previously worked in different projects/faculties and now they have all been brought together.So this is where I come in to help dictate the application development process by designing and maintaining the infrastructure to build an efficient route to live regardless of the project. So this is essentially a blank slate.

As a very high level overview

So my ideal setup would be creating a production environment for whatever project, replicating the stack and using that as UAT. This would be done through Terraform and Ansible. The Dev estate I am not too sure how this is going to be setup yet but I was thinking about learning docker and k8s to bring up a dev estate.

The way I would look into setting up a CI/CD pipeline using Jenknnis and actioned through Azure DevOps as this is what they use, I would ensure code pushed from the DEV branch is deployed to UAT and tested then deployed to replicate production servers and once tested DNS is switched over to the new servers and the old ones are brought down.

The way how the code is deployed is through building the codebase on the same server Jenkins would be setup on, sending a tar'd file of the code to AWS CodeDeploy to be taken to the next environment.

Through caveats include no changes or access will be allowed on production or UAT servers, any requested changes will be done through terraform and ansible to keep things consistent.

Personally with my limited experience I have used Terraform and ansible and I have felt this provides alot of the work to have that infrastructure as code system in place. At the same time I understand there are tools like chef, puppet and salt which would probably make the job easier but that is where my question is what would you guys want and how would you want to implement it. I should finally mention we seem to be using Azure DevOps as the main tool to setup and manage the pipelines.

**Tags:** amazon-web-services, ansible, terraform, azure-devops

> **Comments:**
> **username_1**: Could you make this question more generic? Please process the feedback as provided in one of the answers by BruceBecker. Once done, please leave a comment so this question can be evaluated again.  

---

## 2 answers

### Answer by username_4 on 2019-11-06. Score: 5 | Accepted answer

This question runs two risks:

1. of being marked as "primarily opinion-based", and
2. its answers might age poorly.

However, I think it's useful to express an opinion, if only for posterity; if and when things change, there may be better answers -- but there will always be blank slates.

I would split the problem up into four sections:

1. Infrastructure
2. Application Building, testing and integration
3. Deployment
4. Monitoring, logging and analytics

Furthermore, let's make this cloud-independent (so, this toolkit should be valid for any or all of AWS, GCE, Azure *etc*, as well as on-prem OpenStack, bare metal cloud (e.g. [Packet](https://www.packet.com/), etc).

## Infrastructure

Infrastructure components consist of

1. Your environments (pre, prod, integration, testing, etc - however you define them). These should be built with Terraform modules.

   1. Each module in its own repo
   2. Consider carefully separation of state, so that you can easily create and destroy components and keep costs down
2. "DevOps" infrastructure. This includes:

   1. code repos
   2. CI/CD instances (controller, agent pools, *etc*) - *e.g.* Jenkins master with cloud agent pools.
   3. Deployment runners (see later).
   4. Monitoring, logging and analytics.
      Again, build these with terraform modules. As a example, have a Terraform module for each type of application or component.
3. Team, messaging and support infrastructure. This is often overlooked, but choosing the right tool for the job on Day 2 is hard. It's better to communicate to the stakeholders how issues and change requests should be made. I would venture a combination of GitHub issues and Slack at the very least.
4. secrets management. You will need to pass around a lot of secrets, if you want to foster collaboration. In order to do this securely, you will need a place to manage them and delegate access to and usage of them.

## Building, testing, integration.

These happen in Jenkins. Each application should contain a subdirectory `.pipeline` with the parts necessary to build it. The part which concerns us here is everything before prod.

I would suggest a pipeline as code approach with a `Jenkinsfile` in the repo declaring the steps for

1. building the application:
2. Testing the application
   1. Unit tests
   2. Functional tests
   3. Integration tests
   4. (Regression tests)
3. Building artifacts
   1. building and signing artifacts
   2. Shipping them to the artifactory, with new version info and other metadata
4. Building instance images (if your application is "installed" in an environment, and not serverless).
5. Enforcing changes to the Infrastructure (terraform plan/apply)
6. Enforcing changes to the deployment configuration (capacity, *etc*)

At the end of this stage, you'll have a properly configured environment (say, pre or prod), an artifact, and an image which you can deploy. If need be, you can run performance tests (with Taurus) or acceptance tests (with cucumber) on them with a job in Jenkins.

## Deployment

I would vote for a hard separation between the previous phase and this one. Deployment should be done with a different tool and the right tool for the job is in my opinion Spinnaker. The configuration for the deployment should be kept in the same repo as the application (so that developers can serve themselves). Deployments should be done on every change to the codebase, declaratively -- Spinnaker has a JSON syntax for this, making it easy to check consistency and correctness of the deployment scenario.

## Monitoring, logging and analytics

As mentioned before, Day 1 is a blank slate, but it is quickly dirtied. The DevOps doesn't stop on Day 2 (ie, when apps are deployed), and it would be wise to plan ahead.

The SRE book gives [good advice for instrumenting and monitoring the applications](https://landing.google.com/sre/sre-book/chapters/service-level-objectives/) you will be deploying, but this space is very crowded with services and tools claiming solve your monitoring and alerting needs. Again going with the principle that we should be able to deploy anywhere, I would go with a Prometheus for indicator and metric extraction and Grafana for displaying setup. However, you should definitely consider this part wisely.

---

So - the toolkit looks like:

1. [Vault](https://www.vaultproject.io/) for secrets
2. [GitHub](https://github.com/) for code, projects, tasks and issues
3. [Slack](https://slack.com) for communication
4. [Terraform](https://terraform.io) for infrastructure
5. [Ansible](https://www.ansible.com) for configuration management
6. [Packer](https://packer.io) for image baking
7. [Quay](https://quay.io) for (docker) image registry
8. [Inspec](https://inspec.io) for compliance and correctness of state
9. [Cucumber](https://cucumber.io/) for Acceptance Testing
10. [Jenkins](https://jenkins.io) for pipeline execution
11. [Spinnaker](https://www.spinnaker.io/) for Deployment
12. [Prometheus](https://prometheus.io) for metrics [Grafana](https://grafana.com/) for visualisation and dashboards
13. [Taurus](http://gettaurus.org/) (Blazetool) for performance testing

Is this a lot of tools? Yes.

Will you be able to deploy anywhere? Also yes.

Will you need **all** of them? probably not. Also, I've chosen tools which are mostly free to use and open source.

**Also note:** in this scenario, the entire state of all applications can and should be represented as code, and can thus have semantic versioning (or other kind of versioning which makes sense to you). Any changes to that state can be represented by changes to the code, and can thus have good source code management principles applied to it, such as code review, automated testing, *etc*.

> **Comments:**
> **username_0**: Thank you for the highly detailed answer. I think the packet tool sounds quite promising. The main brain picker really is how development is going to stay consistant as possible between projects and in line with UAT. The plan as mentioned is to keep UAT and PROD the same and only change with deployments to ensure no issues can come about. I can see myself using packer to deploy set images based on requirements so if issues were to come about that could help replicate a bug along side the dev instances if they are developing on the same packet image they are pulling from if that makes sense?  
> **username_4**: We follow the pattern of re-using the Ansible roles, and setting environment-specific variables, to change configuration. The playbooks can then largely be re-used and the Ansible roles independently tested. In packer, we typically build all environments at the same time, so that they are linked back to a git commit.  
> **username_0**: Not sure i understand your wording here, are you saying you use ansible to build the consistent enviorments and packet is used to build them at the same time?  
> **username_4**: (note : Packer is a tool from Hashicorp to build images. Packet.net is a bare-metal cloud provider -- these are different :) ). I'm afraid the mods will shut us down here for using the comments to chat, so perhaps it would be best to ask a different question to clarify this.  
> **username_0**: Its okay i understand it now, so as a high level overview i could use terraform to provision the servers with packer which basically installs things like nginx and stuff before the server is actually deployed then can use ansible actually setup the specifics.  

---

### Answer by username_2 on 2019-11-06. Score: 1

I am a big fan of saltstack, mainly because it provides a comprehensive solution regarding the whole server life cycle, unlike ansible which need another tool (such as terraform) to provision vms.

salt + salt-cloud is one very good suite from my opinion.

I do not see any differences regarding the amount of work required by either tool (chef, puppet, ansible, salt) to get a working IAC system in place, this is, of course, according to your language preference.
If you're in python then go for salt or ansible, otherwise go check puppet or chef.
Do not forget to plug your iac code base to a VCS.

> **Comments:**
> **username_3**: The push (salt/ansible) vs pull (chef/puppet) model difference may play a role in the choice too (for DMZ systems for example)  
> **username_2**: salt has an interesting ability for both model,  pull (salt-call executed from the minion) and push (salt command executed on the master), this helps a lot for minion with an unstable connection.  
> **username_3**: Missed that model in salt, I didn't check it for a while that said, it still use the system python (as ansible) if I get it right, which make it quite complicated when running legacy apps incompatible with its requirements (Chef has a point here coming with its own ruby independent of the system one at the cost of a larger client of course)  

---

