<p>We're using jobs in jenkins to trigger deployments. It ensures that no matter who does the deployment, the ansible command that is run will be the same. A nice bonus is the build logs record when deployments were triggered, who triggered them and what exactly happened during the deployment.</p>

<p>It's certainly not foolproof, but it's been a nice improvement over running ansible playbooks by hand.</p>

<p>For larger/riskier changes this should ideally be combined with some form of change management so changes are made only after another person/team reviews the change and the approach to the change to help identify and resolve potential issues early.</p>

<p>Additionally it never hurts to have a teammate who understands the change you're making be present and watching while you make big changes so they can watch for and help prevent mistakes in the execution of the change.</p>

CC BY-SA 3.0
2017-03-04T20:43:33.097Z
2017-03-04T20:43:33.097Z
bradym
<p><strong>Categories of errors</strong></p>

<p>There are two ways of looking at human factors that lead to problems and accidents:</p>

<ol>
<li>You can see the human error as the cause of a mishap. In this case
"human error," under whatever label—loss of situation awareness, procedural violation, regulatory shortfalls, managerial deficiencies is the conclusion of your investigation.</li>
<li>You can see the human error as the symptom of deeper trouble. In this case, human error is the starting point for your investigation. You
will probe how human error is systematically connected to features
of people's tools, tasks, and operational / organizational environment.</li>
</ol>

<p>The first is called the <em>human</em> approach, and the second as the <em>system</em> approach.</p>

<p>To explain failure using the human approach, you would seek failure and find people's inaccurate assessments, wrong decisions or bad judgments.</p>

<p>To explain failure using the system approach, you are not trying to find where people went wrong. Instead,  find how people's assessments and actions made sense at the time, given the circumstances that surrounded them.</p>

<p>For example, Donald Berwick of the Institute for Healthcare Improvement (IHI) argues that improving patient safety requires <em>changes in the design of systems</em>:</p>

<blockquote>
  <p>...We are human, and humans err. Despite outrage, despite grief, despite experience, despite our best efforts, despite our deepest wishes, we are born fallible and will remain so. Being careful helps, but it brings us nowhere near perfection... The remedy is in changing systems of work. The remedy is in design. The goal should be extreme safety. I believe we should be as safe in our hospitals as we are in our homes. But we cannot reach that goal through exhortation, censure, outrage, and shame. We can reach it only by a commitment to change, so that normal, human errors can be made irrelevant to the outcome, continually found, and skillfully mitigated.</p>
  
  <p><a href="http://www.bmj.com/content/322/7281/247" rel="nofollow noreferrer">Donald M Berwick. Not again! BMJ 2001</a></p>
</blockquote>

<hr>

<p><strong>Removing mistakes from the system</strong></p>

<p>A great way to find (and correct) the various ways failure happens after the fact, is searching for the root cause without blaming the people. This is often called "blameless post-mortems", and <a href="https://codeascraft.com/2012/05/22/blameless-postmortems/" rel="nofollow noreferrer">Etsy Code as Craft blog post</a> expands on the concept. People at Etsy presented and wrote <a href="https://www.infoq.com/articles/postmortems-etsy" rel="nofollow noreferrer">more about it on other forums</a> and blogs.</p>

<p>To prevent mistakes in the first place, some culture traits are a must. Procedures and various artifacts created in the system must test that using them by humans is very clear and self-explanatory. Often those who create are not those who consume, leading to a disconnect and lack of clarity. The system is then not safe to operate since the only person who knows all the assumptions is the one who created it (and no one else).</p>

<p><em>Effective control measures</em></p>

<p>Put in place effective control measures to stop the process when an error occurs. This is mistake-proofing. Effective control measures are design changes that prevent or stop processes from continuing when an error has occurred by introducing a process failure</p>

<p>Example:</p>

<p>In 1896, Sakichi Toyoda invented Japan’s first power loom called “the Toyoda Steam power loom.” This development increased productivity by twenty times, and the quality of the textiles improved and caused a revolution in the textile industry in Japan. But here’s the subtle but very important discovery and principle:</p>

<blockquote>
  <p>when the needle broke, the machine stopped</p>
</blockquote>

<p>Sakichi Toyoda created an innovation to the Loom that would later become one of the pillars in the Toyota Production System (Lean). That pillar we now call Jidoka, sometimes called “smart automation with a human touch” or “autonomation.”</p>

<p>In large part, Andon (stop at first defect) and Poka-Yoke (mistake proofing) are later developments that find their influence from the Loom.</p>

<p><em>Removing Single-Point Weaknesses</em></p>

<p>The term single-point weakness refers to the creation of redundancies in the system as an approach to improving system reliability. Redundancy is created by increasing the number of systems or individuals involved in the process. Having more backup systems or more checks (double, triple, or more) increases the probability that the process will proceed correctly.</p>

<p>One great example for this is the "four-eyes principle," which means that "all business decisions and transactions need approval from the CEO and CFO. Since the CFO is not reporting to the CEO, there is an independent controlling mechanism in place".</p>

<p><sup>source: <a href="https://en.wikipedia.org/wiki/Two-man_rule" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Two-man_rule</a></sup></p>

<p><em>Make Hazards Obvious</em></p>

<p>If hazards are made obvious, or impossible to reach, humans cannot create mistakes. For example, color-coding is a common approach to making mistakes more obvious. Or if you think of various computer sockets that can only be inserted one way and not the other, etc.</p>

<hr>

<p>Some great books are talking about the subject, and it would not be a good answer without mentioning them:</p>

<ul>
<li><a href="http://rads.stackoverflow.com/amzn/click/B00Q8XCSFI" rel="nofollow noreferrer">The Field Guide to Understanding 'Human Error'</a> by Sidney Dekker</li>
<li><a href="http://rads.stackoverflow.com/amzn/click/B01DCPXKZ6" rel="nofollow noreferrer">Site Reliability Engineering: How Google Runs Production Systems</a></li>
<li><a href="http://rads.stackoverflow.com/amzn/click/0915299070" rel="nofollow noreferrer">Zero Quality Control: Source Inspection and the Poka-Yoke System</a> by Shigeo Shingo</li>
</ul>

CC BY-SA 3.0
2017-03-04T11:27:50.563Z
A very important method you do not mention is the “four-eyes principle” which is used in the finance – either as a regulatory obligation or as a safe-guard. In the software industry it is implemented in various ways, as e.g. code reviews but also can be used to validate commands affecting live systems.
Michaël Le Barbier
CC BY-SA 3.0
2017-03-04T11:29:04.550Z
I'll add that to the SPW principle.
Evgeny Zislis
CC BY-SA 3.0
2017-03-04T13:03:59.107Z
Great discussion about errors, but it doesn't say how to safeguard against accidental deployments.
Alexandre
CC BY-SA 3.0
2017-03-05T00:13:13.667Z
The question asks about Ansible specifically. This answer is very thorough and well-researched but it is one step removed from the real-world problem.
Woodland
CC BY-SA 3.0
2017-03-05T06:39:32.647Z
@Evgeny When I replied to your AWS Lambda performance question, at first I didn't say *how* to run your tests and you pointed that out. You were right, and I adjusted my answer. I understand people that are down voting your answer here. Your answer would be good for a question about "How to approach and reduce errors in our workplace?". Here, OP has a question about Ansible and you don't even mention it. Worse, OP gives an indication on the kind of solution he's looking for, and you are going the other way. Your answer is *great* (really), but not for this question.
Alexandre
CC BY-SA 3.0
2017-03-05T07:06:55.913Z
@Alexandre ok. I think I read something else that OP meant, that some other people didn't read the same way as me. I'll leave it to the OP to edit (or not edit) his question with regards to this answer.
Evgeny Zislis
CC BY-SA 3.0
2017-03-04T11:22:25.333Z
2017-03-04T20:53:06.707Z
2017-03-04T20:53:06.707Z
Evgeny Zislis
<p>As @bradim said using your CI/CD tool to initiate the deployment instead of hand based commands is usually good step forward, as is adding tests in your pipeline that actually test your deployment scripts on your staging (or a freshly created) environment, where you can pick up bugs earlier.</p>

<p>I would also add that instead of calling your ansible scripts directly, you can also add tools like <a href="https://www.ansible.com/tower" rel="nofollow noreferrer">Ansible Tower</a> into your flow, which will let you track the changes that have been run more easily, and can give you an additional step of security into your flow.</p>

CC BY-SA 3.0
2017-03-06T10:28:15.623Z
2017-03-06T10:28:15.623Z
SztupY
<p>Recently the Amazon S3 had a <a href="https://aws.amazon.com/message/41926/" rel="noreferrer">major outage</a> in the us-east-1 region. It looks like it was likely caused by a spelling error when running a maintenance playbook in Ansible or a similar tool. You can put a shell script wrapper around ansible-playbook to look like:</p>

<pre><code>#!/bin/bash
/usr/bin/ansible-playbook "$@" --list-hosts --list-tasks
read -p "Are you sure? (y/n) " answer
test "$answer" = "y" || exit 0
exec /usr/bin/ansible-playbook "$@"
</code></pre>

<p>But what are some other ways you use to improve the safety and reduce a chance of error causing a major outage for your company.</p>

CC BY-SA 3.0
2017-03-04T06:55:40.837Z
I'm voting to close this question as off-topic because it will be more suitable for http://unix.stackexchange.com/ or http://superuser.com/
Romeo Ninov
CC BY-SA 3.0
2017-03-04T07:02:49.933Z
Infrastructure as code, is one of the key components to get to hundreds of deployments a day. Being able to secure the tools that provide this speed from creating major outage in the operations seems to me like a relevant topic. I might be wrong, of course. I do appreciate your view though. Would you like to join this discussion about on and off topic questions in Meta?
Jiri Klouda
CC BY-SA 3.0
2017-03-04T07:09:58.267Z
For example this question seems to be accepted as on topic: http://devops.stackexchange.com/questions/98/how-to-test-provisioning-and-configuration-in-ansible-setup
Jiri Klouda
CC BY-SA 3.0
2017-03-04T07:16:44.273Z
Jiri, do you make difference between your and other question you mention?
Romeo Ninov
CC BY-SA 3.0
2017-03-04T08:57:14.923Z
If these kinds of questions would be suitable for superuser, there would be no need for devops.se. This is definitely on topic here. Operations and mitigating human error are in the core of DevOps.
Evgeny Zislis
CC BY-SA 3.0
2017-03-05T05:01:20.303Z
@RomeoNinov one of the reason for creating DevOps was exactly to have a one-stop shop for all kinds of DevOps-related questions instead of hopping across a plethora of SE sites: http://discuss.area51.stackexchange.com/questions/22356/isnt-duplicate-of-server-fault
Dan Cornilescu
<ansible><process>
How to safeguard Ansible deployment to mitigate accidents?
Jiri Klouda
# How to safeguard Ansible deployment to mitigate accidents?

(Asked by: username_0 on 2017-03-04)

Recently the Amazon S3 had a major outage in the us-east-1 region. It looks like it was likely caused by a spelling error when running a maintenance playbook in Ansible or a similar tool. You can put a shell script wrapper around ansible-playbook to look like:
#!/bin/bash
/usr/bin/ansible-playbook "$@" --list-hosts --list-tasks
read -p "Are you sure? (y/n) " answer
test "$answer" = "y" || exit 0
exec /usr/bin/ansible-playbook "$@"

But what are some other ways you use to improve the safety and reduce a chance of error causing a major outage for your company.

**Tags:** ansible, process

> **Comments:**
> **username_1**: I'm voting to close this question as off-topic because it will be more suitable for http://unix.stackexchange.com/ or http://superuser.com/  
> **username_0**: Infrastructure as code, is one of the key components to get to hundreds of deployments a day. Being able to secure the tools that provide this speed from creating major outage in the operations seems to me like a relevant topic. I might be wrong, of course. I do appreciate your view though. Would you like to join this discussion about on and off topic questions in Meta?  
> **username_0**: For example this question seems to be accepted as on topic: http://devops.stackexchange.com/questions/98/how-to-test-provisioning-and-configuration-in-ansible-setup  
> **username_1**: Jiri, do you make difference between your and other question you mention?  
> **username_2**: If these kinds of questions would be suitable for superuser, there would be no need for devops.se. This is definitely on topic here. Operations and mitigating human error are in the core of DevOps.  
> **username_3**: @RomeoNinov one of the reason for creating DevOps was exactly to have a one-stop shop for all kinds of DevOps-related questions instead of hopping across a plethora of SE sites: http://discuss.area51.stackexchange.com/questions/22356/isnt-duplicate-of-server-fault  

---

## 3 answers

### Answer by username_4 on 2017-03-04. Score: 6

We're using jobs in jenkins to trigger deployments. It ensures that no matter who does the deployment, the ansible command that is run will be the same. A nice bonus is the build logs record when deployments were triggered, who triggered them and what exactly happened during the deployment.

It's certainly not foolproof, but it's been a nice improvement over running ansible playbooks by hand.

For larger/riskier changes this should ideally be combined with some form of change management so changes are made only after another person/team reviews the change and the approach to the change to help identify and resolve potential issues early.

Additionally it never hurts to have a teammate who understands the change you're making be present and watching while you make big changes so they can watch for and help prevent mistakes in the execution of the change.


---

### Answer by username_2 on 2017-03-04. Score: 4

**Categories of errors**

There are two ways of looking at human factors that lead to problems and accidents:

1. You can see the human error as the cause of a mishap. In this case
   "human error," under whatever label—loss of situation awareness, procedural violation, regulatory shortfalls, managerial deficiencies is the conclusion of your investigation.
2. You can see the human error as the symptom of deeper trouble. In this case, human error is the starting point for your investigation. You
   will probe how human error is systematically connected to features
   of people's tools, tasks, and operational / organizational environment.

The first is called the *human* approach, and the second as the *system* approach.

To explain failure using the human approach, you would seek failure and find people's inaccurate assessments, wrong decisions or bad judgments.

To explain failure using the system approach, you are not trying to find where people went wrong. Instead, find how people's assessments and actions made sense at the time, given the circumstances that surrounded them.

For example, Donald Berwick of the Institute for Healthcare Improvement (IHI) argues that improving patient safety requires *changes in the design of systems*:

> ...We are human, and humans err. Despite outrage, despite grief, despite experience, despite our best efforts, despite our deepest wishes, we are born fallible and will remain so. Being careful helps, but it brings us nowhere near perfection... The remedy is in changing systems of work. The remedy is in design. The goal should be extreme safety. I believe we should be as safe in our hospitals as we are in our homes. But we cannot reach that goal through exhortation, censure, outrage, and shame. We can reach it only by a commitment to change, so that normal, human errors can be made irrelevant to the outcome, continually found, and skillfully mitigated.
>
> [Donald M Berwick. Not again! BMJ 2001](http://www.bmj.com/content/322/7281/247)

---

**Removing mistakes from the system**

A great way to find (and correct) the various ways failure happens after the fact, is searching for the root cause without blaming the people. This is often called "blameless post-mortems", and [Etsy Code as Craft blog post](https://codeascraft.com/2012/05/22/blameless-postmortems/) expands on the concept. People at Etsy presented and wrote [more about it on other forums](https://www.infoq.com/articles/postmortems-etsy) and blogs.

To prevent mistakes in the first place, some culture traits are a must. Procedures and various artifacts created in the system must test that using them by humans is very clear and self-explanatory. Often those who create are not those who consume, leading to a disconnect and lack of clarity. The system is then not safe to operate since the only person who knows all the assumptions is the one who created it (and no one else).

*Effective control measures*

Put in place effective control measures to stop the process when an error occurs. This is mistake-proofing. Effective control measures are design changes that prevent or stop processes from continuing when an error has occurred by introducing a process failure

Example:

In 1896, Sakichi Toyoda invented Japan’s first power loom called “the Toyoda Steam power loom.” This development increased productivity by twenty times, and the quality of the textiles improved and caused a revolution in the textile industry in Japan. But here’s the subtle but very important discovery and principle:

> when the needle broke, the machine stopped

Sakichi Toyoda created an innovation to the Loom that would later become one of the pillars in the Toyota Production System (Lean). That pillar we now call Jidoka, sometimes called “smart automation with a human touch” or “autonomation.”

In large part, Andon (stop at first defect) and Poka-Yoke (mistake proofing) are later developments that find their influence from the Loom.

*Removing Single-Point Weaknesses*

The term single-point weakness refers to the creation of redundancies in the system as an approach to improving system reliability. Redundancy is created by increasing the number of systems or individuals involved in the process. Having more backup systems or more checks (double, triple, or more) increases the probability that the process will proceed correctly.

One great example for this is the "four-eyes principle," which means that "all business decisions and transactions need approval from the CEO and CFO. Since the CFO is not reporting to the CEO, there is an independent controlling mechanism in place".

source: <https://en.wikipedia.org/wiki/Two-man_rule>

*Make Hazards Obvious*

If hazards are made obvious, or impossible to reach, humans cannot create mistakes. For example, color-coding is a common approach to making mistakes more obvious. Or if you think of various computer sockets that can only be inserted one way and not the other, etc.

---

Some great books are talking about the subject, and it would not be a good answer without mentioning them:

* [The Field Guide to Understanding 'Human Error'](http://rads.stackoverflow.com/amzn/click/B00Q8XCSFI) by Sidney Dekker
* [Site Reliability Engineering: How Google Runs Production Systems](http://rads.stackoverflow.com/amzn/click/B01DCPXKZ6)
* [Zero Quality Control: Source Inspection and the Poka-Yoke System](http://rads.stackoverflow.com/amzn/click/0915299070) by Shigeo Shingo

> **Comments:**
> **username_5**: A very important method you do not mention is the “four-eyes principle” which is used in the finance – either as a regulatory obligation or as a safe-guard. In the software industry it is implemented in various ways, as e.g. code reviews but also can be used to validate commands affecting live systems.  
> **username_2**: I'll add that to the SPW principle.  
> **username_6**: Great discussion about errors, but it doesn't say how to safeguard against accidental deployments.  
> **username_7**: The question asks about Ansible specifically. This answer is very thorough and well-researched but it is one step removed from the real-world problem.  
> **username_6**: @Evgeny When I replied to your AWS Lambda performance question, at first I didn't say *how* to run your tests and you pointed that out. You were right, and I adjusted my answer. I understand people that are down voting your answer here. Your answer would be good for a question about "How to approach and reduce errors in our workplace?". Here, OP has a question about Ansible and you don't even mention it. Worse, OP gives an indication on the kind of solution he's looking for, and you are going the other way. Your answer is *great* (really), but not for this question.  
> **username_2**: @username_6 ok. I think I read something else that OP meant, that some other people didn't read the same way as me. I'll leave it to the OP to edit (or not edit) his question with regards to this answer.  

---

### Answer by username_8 on 2017-03-06. Score: 1

As @bradim said using your CI/CD tool to initiate the deployment instead of hand based commands is usually good step forward, as is adding tests in your pipeline that actually test your deployment scripts on your staging (or a freshly created) environment, where you can pick up bugs earlier.

I would also add that instead of calling your ansible scripts directly, you can also add tools like [Ansible Tower](https://www.ansible.com/tower) into your flow, which will let you track the changes that have been run more easily, and can give you an additional step of security into your flow.


---

