<blockquote>
  <p><strong>NOTE:</strong> It's probably not worth reading too much into the outward comprehension of how good-or-not StackExchange is at managing their disaster recovery scenarios.  I suspect they are following much of the best practice below and simply testing scenarios to validate their configuration.</p>
</blockquote>

<p>Depending on the environment you operate within:</p>

<ul>
<li><p>A disaster recovery plan might form part of a larger <a href="https://www.ready.gov/business/implementation/continuity" rel="noreferrer">Business Continuity Plan</a>, a Business Continuity Plan would also likely consider operational risks to you people, organisation, locations, information, partners and management systems.</p></li>
<li><p>A disaster recovery plan might be broken down into many IT Service Continuity Plans for individual services.  The disaster recovery plan would likely incorporate people and process along with the technical aspects of the service.
<a href="https://i.sstatic.net/2wf2m.png" rel="noreferrer"><img src="https://i.sstatic.net/2wf2m.png" alt="BCP Hierarchy"></a></p></li>
</ul>

<p>Given these definitions, you might consider ways of improving the capability of the organisation as a whole to be resilient against failure:</p>

<ul>
<li><p><strong>Service Recovery</strong>:</p>

<ul>
<li>Enable individual services to be <a href="https://infocus.emc.com/john_linse/what-is-activeactive-and-how-can-it-improve-business-continuity/" rel="noreferrer">Active-Active</a> across two geographically dispersed data centres.  This does assume that applications are capable of replicating state between data centres, for example using <a href="https://en.wikipedia.org/wiki/Eventual_consistency" rel="noreferrer">BASE Semantics</a> for data.</li>
<li>Create Self-Healing services, this means expecting failure and building with <a href="http://erikhollnagel.com/ideas/resilience-engineering.html" rel="noreferrer">Resilience Engineering</a> in mind.  An example is by using a tool such as <a href="https://devops.stackexchange.com/q/609/397">Chaos Monkey</a> to simulate a failure.</li>
</ul></li>
<li><p><strong>Disaster Recovery Plan</strong>:</p>

<ul>
<li>Again enable <a href="https://infocus.emc.com/john_linse/what-is-activeactive-and-how-can-it-improve-business-continuity/" rel="noreferrer">Active-Active</a> mechanisms across data centres, the difference from SRPs is that capacity needs to be carefully considered, i.e. if you have to DCs in an Active-Active pattern and one fails then a single DC must be sufficiently scaled to support 100% of the traffic.</li>
<li>War Games and Rehearsals are really important for disaster recovery plans as this tests the people and the process, in the most mature DevOps environments much of this can be automated as evidenced by <a href="http://techblog.netflix.com/2011/07/netflix-simian-army.html" rel="noreferrer">Chaos Gorilla</a>.</li>
</ul></li>
<li><p><strong>Business Continuity Plan</strong>:</p>

<ul>
<li>On the basis that this is a DevOps site I won't go into the long process of building Business Continuity plans.  However, the rules of not keeping all of your eggs in one basket apply - have a plan for when your office is flooded:

<ul>
<li>Allow your staff to work remotely from home one day a week, this tests your BCP strategies.</li>
<li>If possible have to geographically and politically separate locations for your workforce.</li>
<li>Define and test a clear process for communicating business continuity events and practice them through fire drills.</li>
</ul></li>
</ul></li>
</ul>

CC BY-SA 3.0
2017-04-29T09:10:12.413Z
Great answer Richard! And you did include (1) the Monkey (2) the Gorilla and (3) the Eggs that I thought about right away when I first read the question I quoted (thinking something then like "Hm, does SE use Monkeys or Gorillas, and where do they put their Eggs?". So therefor: +1, and accept ... though my "accept" may change at any time, eg if [this user](https://devops.meta.stackexchange.com/users/1001) ever beats your answer ...
Pierre.Vriens
CC BY-SA 3.0
2017-05-04T05:54:10.540Z
How come sites like facebook won't be affected by planned maintainence
TheGameiswar
CC BY-SA 3.0
2017-04-29T00:29:25.233Z
2017-04-29T00:29:25.233Z
Richard Slater
<p>Have a look at what's described in the question about "<a href="https://meta.stackexchange.com/q/295285/286501">Brief outage planned for Wed, May 3, 2017 at 0:00 UTC, 8pm US/Eastern (like a fire drill for computers)</a>", which is about testing of the existing "<strong>Disaster Recovery Planning</strong>" (= DRP) of the entire family of SE sites.</p>

<p>If you were in charge of this, what would be your recommendations to improve these kinds of DRP-testing in production?</p>

CC BY-SA 3.0
2017-04-28T10:02:53.943Z
@Dawny33 indeed, as in my suggested tag excerpt, right? BTW, IMO it's a good practise to submit, right away, an edit suggestion for a newly created tag ... by the user who first introduced the tag ... That way you avoid the risk it gets abused for (eg) "**D**on't **R**uine **P**roduction" (whatever that may mean ...) ...
Pierre.Vriens
CC BY-SA 3.0
2017-04-28T19:19:43.007Z
@Pierre.Vriens I changed the tag to use the full name rather than the abbreviation; it's within the tag length limit, and so I think it's preferable to have the full name where possible.
Aurora0001
CC BY-SA 3.0
2017-04-28T19:31:14.817Z
@Aurora0001: fine for me (I actually doubted when I first posted my question).
Pierre.Vriens
CC BY-SA 3.0
2017-04-29T08:57:24.103Z
@Aurora0001 : how about the **roi** tag, same thing you think?
Pierre.Vriens
CC BY-SA 3.0
2017-04-29T08:59:28.873Z
@Pierre.Vriens Yeah, I'd say so. If the abbreviation is popular, they could also be synonyms.
Aurora0001
<testing><disaster-recovery-plans>
How to improve DRP-testing of SE sites?
Pierre.Vriens
# How to improve DRP-testing of SE sites?

(Asked by: username_0 on 2017-04-28)

Have a look at what's described in the question about "[Brief outage planned for Wed, May 3, 2017 at 0:00 UTC, 8pm US/Eastern (like a fire drill for computers)](https://meta.stackexchange.com/q/295285/286501)", which is about testing of the existing "**Disaster Recovery Planning**" (= DRP) of the entire family of SE sites.

If you were in charge of this, what would be your recommendations to improve these kinds of DRP-testing in production?

**Tags:** testing, disaster-recovery-plans

> **Comments:**
> **username_0**: @Dawny33 indeed, as in my suggested tag excerpt, right? BTW, IMO it's a good practise to submit, right away, an edit suggestion for a newly created tag ... by the user who first introduced the tag ... That way you avoid the risk it gets abused for (eg) "**D**on't **R**uine **P**roduction" (whatever that may mean ...) ...  
> **username_1**: @username_0 I changed the tag to use the full name rather than the abbreviation; it's within the tag length limit, and so I think it's preferable to have the full name where possible.  
> **username_0**: @username_1: fine for me (I actually doubted when I first posted my question).  
> **username_0**: @username_1 : how about the **roi** tag, same thing you think?  
> **username_1**: @username_0 Yeah, I'd say so. If the abbreviation is popular, they could also be synonyms.  

---

## 1 answers

### Answer by username_2 on 2017-04-29. Score: 7 | Accepted answer

> **NOTE:** It's probably not worth reading too much into the outward comprehension of how good-or-not StackExchange is at managing their disaster recovery scenarios. I suspect they are following much of the best practice below and simply testing scenarios to validate their configuration.

Depending on the environment you operate within:

* A disaster recovery plan might form part of a larger [Business Continuity Plan](https://www.ready.gov/business/implementation/continuity), a Business Continuity Plan would also likely consider operational risks to you people, organisation, locations, information, partners and management systems.
* A disaster recovery plan might be broken down into many IT Service Continuity Plans for individual services. The disaster recovery plan would likely incorporate people and process along with the technical aspects of the service.
  [![BCP Hierarchy](https://i.sstatic.net/2wf2m.png)](https://i.sstatic.net/2wf2m.png)

Given these definitions, you might consider ways of improving the capability of the organisation as a whole to be resilient against failure:

* **Service Recovery**:

  + Enable individual services to be [Active-Active](https://infocus.emc.com/john_linse/what-is-activeactive-and-how-can-it-improve-business-continuity/) across two geographically dispersed data centres. This does assume that applications are capable of replicating state between data centres, for example using [BASE Semantics](https://en.wikipedia.org/wiki/Eventual_consistency) for data.
  + Create Self-Healing services, this means expecting failure and building with [Resilience Engineering](http://erikhollnagel.com/ideas/resilience-engineering.html) in mind. An example is by using a tool such as [Chaos Monkey](https://devops.stackexchange.com/q/609/397) to simulate a failure.
* **Disaster Recovery Plan**:

  + Again enable [Active-Active](https://infocus.emc.com/john_linse/what-is-activeactive-and-how-can-it-improve-business-continuity/) mechanisms across data centres, the difference from SRPs is that capacity needs to be carefully considered, i.e. if you have to DCs in an Active-Active pattern and one fails then a single DC must be sufficiently scaled to support 100% of the traffic.
  + War Games and Rehearsals are really important for disaster recovery plans as this tests the people and the process, in the most mature DevOps environments much of this can be automated as evidenced by [Chaos Gorilla](http://techblog.netflix.com/2011/07/netflix-simian-army.html).
* **Business Continuity Plan**:

  + On the basis that this is a DevOps site I won't go into the long process of building Business Continuity plans. However, the rules of not keeping all of your eggs in one basket apply - have a plan for when your office is flooded:
    - Allow your staff to work remotely from home one day a week, this tests your BCP strategies.
    - If possible have to geographically and politically separate locations for your workforce.
    - Define and test a clear process for communicating business continuity events and practice them through fire drills.

> **Comments:**
> **username_0**: Great answer Richard! And you did include (1) the Monkey (2) the Gorilla and (3) the Eggs that I thought about right away when I first read the question I quoted (thinking something then like "Hm, does SE use Monkeys or Gorillas, and where do they put their Eggs?". So therefor: +1, and accept ... though my "accept" may change at any time, eg if [this user](https://devops.meta.stackexchange.com/users/1001) ever beats your answer ...  
> **username_3**: How come sites like facebook won't be affected by planned maintainence  

---

