<p>Sounds like you're talking about a test environment which is constantly re-used without being <strong>reliably</strong> re-initialized for every test execution. This makes such test an unreliable one. Similar, from the reliability perspective, with manual testing, if you want.</p>

<p>IMHO you shouldn't be using such testing <em>inside</em> your CI/CD qualification purposes as that will effectively invalidate your qualification process (at least in that area). Saying that the software passes test X without actually executing test X for every software version delivered or without having the certainty that the <code>pass</code> result obtained is not accidental (due to false positives) will erode your testing's confidence level. False negatives are not credibility damaging, but they are also undesired because of the unnecessary "noise" they create.</p>

<p>It's fine to execute such testing <em>outside</em> your CI/CD qualification process. But you'd be treating a failed result in such testing just like a customer-found bug: you'd need to reliably reproduce the issue to be able to develop a fix for it and confirm that the fix is working. And you can't really do that if the testing is not reliable.</p>

<p>If you plan to address the issue then ideally you'd first develop an automated, reliable test case for reproducing the issue. Which you'd use to develop a fix and confirm its effectiveness (test result should transition from FAIL to PASS). You can (should?) also place this testcase inside your CI/CD qualification process to prevent future re-occurence, if desired - to increase your overall software release quality level.</p>

CC BY-SA 3.0
2017-03-13T18:35:05.987Z
There's a lot to digest in your answer (I'm not sure I get it already completely). But what you wrote about "*execute such testing outside your CI/CD qualification process*": I would expect that the ultimate result of what gets produced/delivered is stored in your QA and prod environments (via CD, either automatic or manual). But that also "*seems*" to me that CI should also deliver its output over there, while "outside" seems like separation or duplication or something, no?
Pierre.Vriens
CC BY-SA 3.0
2017-03-14T02:21:39.640Z
The `inside` and `outside` references are relative to the CI verification loop. Basically I question the reason for the QA environment's existance - most of the tests done in there should be reliable and eventually executed as part of the CI verifications, especially in a continuous deployment context - since you want to execute them on every CI iteration (successful up to that point at least) anyways.
Dan Cornilescu
CC BY-SA 3.0
2017-03-13T14:50:48.267Z
2018-02-27T17:03:02.203Z
2018-02-27T17:03:02.203Z
Dan Cornilescu
<p>The usual approach is to create different environments:</p>

<p>DEV - this is the place where dev team mess the things. Here are create all changes tunings, deploy new version and so on. Here is the place where CI is integrated fully.</p>

<p>PREPROD/QA - this is the place "play" QA/test/validation team do tests. This environment usually freeze during the tests. Integration of CI with this environment is only to provide new version of the product, configurations, etc.</p>

<p>PRODUCTION - is it need to explain :)?</p>

CC BY-SA 3.0
2017-03-13T18:09:30.420Z
ok, that should help to improve stability, merci! My question is about "test" environments, so obviously "production" should not be considered as such. Despite those who use "production" for testing, you know the saying "***The best test is to activate it in production, and if it doesn't work, just perform a rollback/backout!***"?
Pierre.Vriens
CC BY-SA 3.0
2017-03-13T18:14:03.900Z
@Pierre.Vriens, "playing " in prod IMHO is not wise :) Such separation of environment is intentional. On previous job we had 5 different environments.... A votre serivce
Romeo Ninov
CC BY-SA 3.0
2017-03-13T18:22:20.653Z
"I" agree that such playing is ***not*** wise. However what can "you" do about the cowboys (my 'term' I use for such juppies) who keep doing this over and over, and each and every time they get approval from their managers to get around the (eg) monthly release activation, by yet another bugfix (eg for their bugfix from the day before ... which introduced a new bug). You think that doesn't happen in the real world? BTW: about the "freeze" in your answer, you think it makes sense to post a question like "What are sample implementations of a frozen environment?"
Pierre.Vriens
CC BY-SA 3.0
2017-03-13T18:36:26.220Z
@Pierre.Vriens, for me it make perfect sense to post such question. Normally this is regulated by company rules, but devops create quite dynamic environment and this can re a real challenge :)
Romeo Ninov
CC BY-SA 3.0
2017-03-29T20:43:56.683Z
This is my preferred approach, that way it gives an environment where the devs can immediately test their changes in an integrated environment, but keeps QA clean until the the code is ready to be formally tested
Taegost
CC BY-SA 3.0
2017-03-12T20:20:42.557Z
2017-03-12T20:20:42.557Z
Romeo Ninov
<p>To add to Romeo Ninov's answer, internally within an environment you need to try and separate out the applications as much as possible. This is partially why docker has been so successful for dev/test. It let's you almost pretend that you aren't sharing an environment at all.</p>

<p>The other option is to have very clearly defined servers on which the applications run which are separate from the rest of the infrastructure that makes up your environment. Ie. All the environment management or enablement machinery goes on separate, long lived servers. Then you hook in new short lived servers based on a known image to test an application and, if any changes are made to the base image,  you need to apply those changes <strong>everywhere</strong> for every new component. Which means testing the changes against everything.</p>

<p>If an appdev team ask for a change that breaks someone else's application, then tough luck, they need to internally create a mitigant in their code and keep their specific requirements separate from the environments offering.</p>

CC BY-SA 3.0
2017-03-13T18:16:36.987Z
Interesting viewpoints/additions, though there are some things in it that maybe you want to refine/rework: (1) "applications" in this context, what do you mean (some examples?) (2) any idea how this could work in (good old) mainframe environments (3) what's a "mitigant" in this context here? PS: let me know if you think I should create a new question for any of these "things" (bullets).
Pierre.Vriens
CC BY-SA 3.0
2017-03-12T21:41:10.170Z
2017-03-20T15:45:51.243Z
2017-03-20T15:45:51.243Z
hvindin
<p>If you're doing CI/CD, that implies that there are some automated tests happening (CI) prior to deployment (CD). If you're finding a lot of issues in your test environment, that means they aren't being caught by the tests being run prior to deployment; this indicates insufficient automated testing. If the developers are having issues where defects are cropping up in the test environment(s), they need to improve their automated test suites in order to prevent this. This will also improve quality and reliability overall, all the way through into production.</p>

CC BY-SA 3.0
2017-03-13T20:26:29.250Z
2017-03-13T20:26:29.250Z
Adrian
<p>I'll give my experience on this one, mostly because it showcases why some answers are not always applicable.</p>

<p>Some context to start:</p>

<ul>
<li>We have 7 environments to host roughly 80 applications, most of them rely on each others through webservices or shared tables on db2-iSeries.  </li>
<li>For good or bad, the iSeries are our DB system of reference.  </li>
<li>This last point invalidates any idea of bringing the app with its dependencies in an isolated environment as bringing up an AS400 for each would cost too much and we wouldn't have the hardware to run it anyway.</li>
</ul>

<p>What we are doing is not a complete automated Continuous Delivery, we have a schedule of releases to bring up a coherent lot of applications for the general operations. Aside from this each test teams can trigger a release in one of the Q/A environment for the application they are testing and can put a lock on some application version to avoid another team request breaking their tests.</p>

<p>Dependencies of applications are checked before release, so the system won't release something if other applications can't be updated or doesn't match its dependencies needed.
The main idea is to allow updates when it won't impact someone, if there are no tests planned, it should flow from previous environment (and we're aiming at removing the scheduled releases in the 5 firsts environments on mid term now we have validated this 'on demand' method system).</p>

<p>The short version is to have a 'semaphore' system around the applications in the environment, a team should be able to lock its target application with its dependencies (and transitive dependencies) for the time of manual tests.<br>
The implementation of this semaphore is highly dependent on your automation system so I won't extend on that.</p>

<p>Of course the easy way is, as others mentioned, to create a fresh environment for an application with all its dependencies to avoid the semaphore described above.</p>

CC BY-SA 3.0
2017-03-18T09:49:33.773Z
This answer is a variation of what I'm used to (mainframes), where we do these kind of things for at least 1,5 decade or so already (before "DevOps" was born). I wonder if it would make sense to add my own answer here (to further expand on this answer, how we do this with CMN/ZMF for eg "banks"), or just take it to a new (self answered) question. What do you think? Also, I'm curious about that metaphore thing, worth a new question (with reference to this answer)? PS: you mind if I correct some typos?
Pierre.Vriens
CC BY-SA 3.0
2017-03-18T10:11:22.443Z
No problem for edit :) I did keep it generic, That's not much specific to a devops org IMHO. Again DevOps is an organization change, which may help setting up a better automation by sharing the concerns... so I call this a semaphore as in programmation, I don't think it worth a question but that's up to you
Tensibai
CC BY-SA 3.0
2017-03-18T10:20:50.143Z
Ok, edit completed (as usual: rollback / improve as you see fit). BTW, do you have an "s" on your keyboard?!?!?! Apart from that: stuff to think about over the weekend: see my newest meta question ... Bon weekend! Time for gardening over here (pruning ...)
Pierre.Vriens
CC BY-SA 3.0
2017-03-14T12:28:47.580Z
2017-03-18T10:18:58.463Z
2017-03-18T10:18:58.463Z
Tensibai
<p>Assume you're using continuous integration processes which frequently update some target environments, so that every time there are some changes "you" can test your changes right away. That's part of the goals of CI, no?</p>

<p>But, also assume that you have other people involved in your test cycle, e.g. managers or customers. Makes sense to get other people involved in trying to review (break?) your upcoming changes, no?</p>

<p>But if you <strong>continuously</strong> keep delivering changes in the environment in which those other people are, seriously, trying to test them, then multiple issues may arise, such as:</p>

<ul>
<li><strong><code>they</code></strong> might waste their time in reporting issues which, by the time they save the (in depth) report, they cannot even reproduce the issue themselves anymore (e.g because accidently you also ran into the same issue, and already fixed it in their environment).</li>
<li><strong><code>you</code></strong> might not be able to reproduce issues they reported, since the environments in which they ran into some issue, is no longer identical (you (!!!) might have overlayed their environment).</li>
</ul>

<p>So what can you do (how to configure things?) to avoid such (frustrating) situations?</p>

<continuous-integration><testing>
How to avoid continuous integration-caused instabilities in test environments?
Pierre.Vriens
# How to avoid continuous integration-caused instabilities in test environments?

(Asked by: username_0 on 2017-03-12)

Assume you're using continuous integration processes which frequently update some target environments, so that every time there are some changes "you" can test your changes right away. That's part of the goals of CI, no?

But, also assume that you have other people involved in your test cycle, e.g. managers or customers. Makes sense to get other people involved in trying to review (break?) your upcoming changes, no?

But if you **continuously** keep delivering changes in the environment in which those other people are, seriously, trying to test them, then multiple issues may arise, such as:

* **`they`** might waste their time in reporting issues which, by the time they save the (in depth) report, they cannot even reproduce the issue themselves anymore (e.g because accidently you also ran into the same issue, and already fixed it in their environment).
* **`you`** might not be able to reproduce issues they reported, since the environments in which they ran into some issue, is no longer identical (you (!!!) might have overlayed their environment).

So what can you do (how to configure things?) to avoid such (frustrating) situations?

**Tags:** continuous-integration, testing


---

## 5 answers

### Answer by username_6 on 2017-03-14. Score: 10 | Accepted answer

I'll give my experience on this one, mostly because it showcases why some answers are not always applicable.

Some context to start:

* We have 7 environments to host roughly 80 applications, most of them rely on each others through webservices or shared tables on db2-iSeries.
* For good or bad, the iSeries are our DB system of reference.
* This last point invalidates any idea of bringing the app with its dependencies in an isolated environment as bringing up an AS400 for each would cost too much and we wouldn't have the hardware to run it anyway.

What we are doing is not a complete automated Continuous Delivery, we have a schedule of releases to bring up a coherent lot of applications for the general operations. Aside from this each test teams can trigger a release in one of the Q/A environment for the application they are testing and can put a lock on some application version to avoid another team request breaking their tests.

Dependencies of applications are checked before release, so the system won't release something if other applications can't be updated or doesn't match its dependencies needed.
The main idea is to allow updates when it won't impact someone, if there are no tests planned, it should flow from previous environment (and we're aiming at removing the scheduled releases in the 5 firsts environments on mid term now we have validated this 'on demand' method system).

The short version is to have a 'semaphore' system around the applications in the environment, a team should be able to lock its target application with its dependencies (and transitive dependencies) for the time of manual tests.  
The implementation of this semaphore is highly dependent on your automation system so I won't extend on that.

Of course the easy way is, as others mentioned, to create a fresh environment for an application with all its dependencies to avoid the semaphore described above.

> **Comments:**
> **username_0**: This answer is a variation of what I'm used to (mainframes), where we do these kind of things for at least 1,5 decade or so already (before "DevOps" was born). I wonder if it would make sense to add my own answer here (to further expand on this answer, how we do this with CMN/ZMF for eg "banks"), or just take it to a new (self answered) question. What do you think? Also, I'm curious about that metaphore thing, worth a new question (with reference to this answer)? PS: you mind if I correct some typos?  
> **username_6**: No problem for edit :) I did keep it generic, That's not much specific to a devops org IMHO. Again DevOps is an organization change, which may help setting up a better automation by sharing the concerns... so I call this a semaphore as in programmation, I don't think it worth a question but that's up to you  
> **username_0**: Ok, edit completed (as usual: rollback / improve as you see fit). BTW, do you have an "s" on your keyboard?!?!?! Apart from that: stuff to think about over the weekend: see my newest meta question ... Bon weekend! Time for gardening over here (pruning ...)  

---

### Answer by username_1 on 2017-03-13. Score: 8

Sounds like you're talking about a test environment which is constantly re-used without being **reliably** re-initialized for every test execution. This makes such test an unreliable one. Similar, from the reliability perspective, with manual testing, if you want.

IMHO you shouldn't be using such testing *inside* your CI/CD qualification purposes as that will effectively invalidate your qualification process (at least in that area). Saying that the software passes test X without actually executing test X for every software version delivered or without having the certainty that the `pass` result obtained is not accidental (due to false positives) will erode your testing's confidence level. False negatives are not credibility damaging, but they are also undesired because of the unnecessary "noise" they create.

It's fine to execute such testing *outside* your CI/CD qualification process. But you'd be treating a failed result in such testing just like a customer-found bug: you'd need to reliably reproduce the issue to be able to develop a fix for it and confirm that the fix is working. And you can't really do that if the testing is not reliable.

If you plan to address the issue then ideally you'd first develop an automated, reliable test case for reproducing the issue. Which you'd use to develop a fix and confirm its effectiveness (test result should transition from FAIL to PASS). You can (should?) also place this testcase inside your CI/CD qualification process to prevent future re-occurence, if desired - to increase your overall software release quality level.

> **Comments:**
> **username_0**: There's a lot to digest in your answer (I'm not sure I get it already completely). But what you wrote about "*execute such testing outside your CI/CD qualification process*": I would expect that the ultimate result of what gets produced/delivered is stored in your QA and prod environments (via CD, either automatic or manual). But that also "*seems*" to me that CI should also deliver its output over there, while "outside" seems like separation or duplication or something, no?  
> **username_1**: The `inside` and `outside` references are relative to the CI verification loop. Basically I question the reason for the QA environment's existance - most of the tests done in there should be reliable and eventually executed as part of the CI verifications, especially in a continuous deployment context - since you want to execute them on every CI iteration (successful up to that point at least) anyways.  

---

### Answer by username_2 on 2017-03-12. Score: 7

The usual approach is to create different environments:

DEV - this is the place where dev team mess the things. Here are create all changes tunings, deploy new version and so on. Here is the place where CI is integrated fully.

PREPROD/QA - this is the place "play" QA/test/validation team do tests. This environment usually freeze during the tests. Integration of CI with this environment is only to provide new version of the product, configurations, etc.

PRODUCTION - is it need to explain :)?

> **Comments:**
> **username_0**: ok, that should help to improve stability, merci! My question is about "test" environments, so obviously "production" should not be considered as such. Despite those who use "production" for testing, you know the saying "***The best test is to activate it in production, and if it doesn't work, just perform a rollback/backout!***"?  
> **username_2**: @username_0, "playing " in prod IMHO is not wise :) Such separation of environment is intentional. On previous job we had 5 different environments.... A votre serivce  
> **username_0**: "I" agree that such playing is ***not*** wise. However what can "you" do about the cowboys (my 'term' I use for such juppies) who keep doing this over and over, and each and every time they get approval from their managers to get around the (eg) monthly release activation, by yet another bugfix (eg for their bugfix from the day before ... which introduced a new bug). You think that doesn't happen in the real world? BTW: about the "freeze" in your answer, you think it makes sense to post a question like "What are sample implementations of a frozen environment?"  
> **username_2**: @username_0, for me it make perfect sense to post such question. Normally this is regulated by company rules, but devops create quite dynamic environment and this can re a real challenge :)  
> **username_3**: This is my preferred approach, that way it gives an environment where the devs can immediately test their changes in an integrated environment, but keeps QA clean until the the code is ready to be formally tested  

---

### Answer by username_4 on 2017-03-12. Score: 3

To add to username_2's answer, internally within an environment you need to try and separate out the applications as much as possible. This is partially why docker has been so successful for dev/test. It let's you almost pretend that you aren't sharing an environment at all.

The other option is to have very clearly defined servers on which the applications run which are separate from the rest of the infrastructure that makes up your environment. Ie. All the environment management or enablement machinery goes on separate, long lived servers. Then you hook in new short lived servers based on a known image to test an application and, if any changes are made to the base image, you need to apply those changes **everywhere** for every new component. Which means testing the changes against everything.

If an appdev team ask for a change that breaks someone else's application, then tough luck, they need to internally create a mitigant in their code and keep their specific requirements separate from the environments offering.

> **Comments:**
> **username_0**: Interesting viewpoints/additions, though there are some things in it that maybe you want to refine/rework: (1) "applications" in this context, what do you mean (some examples?) (2) any idea how this could work in (good old) mainframe environments (3) what's a "mitigant" in this context here? PS: let me know if you think I should create a new question for any of these "things" (bullets).  

---

### Answer by username_5 on 2017-03-13. Score: 3

If you're doing CI/CD, that implies that there are some automated tests happening (CI) prior to deployment (CD). If you're finding a lot of issues in your test environment, that means they aren't being caught by the tests being run prior to deployment; this indicates insufficient automated testing. If the developers are having issues where defects are cropping up in the test environment(s), they need to improve their automated test suites in order to prevent this. This will also improve quality and reliability overall, all the way through into production.


---

