<p>There's one possibility I can think of, if when the dev are working on their own workstation, with sometimes images baked for virtual box to run on their workstation where your infrastructure doesn't use the exact same image.</p>

<p>The dev will need, while developing a feature, need to add a JVM parameter or whatever change to the middleware early in its work and forget it.</p>

<p>Before commiting, all unit/integration tests run on its workstation works great, as the baked image is shared, it works on every develloper system.</p>

<p>But when going through CI, it fails because the change to the middle-ware wasn't implemented, either because the dev forgot to ask for it, or just because the team in charge of updating the base images/provisioning system didn't had the time or did forget to update the system.</p>

<p>That's a good thing it break in CI, because it tells early before going into production that the system won't work as expected, but sometimes it becomes a hell to find the missing parameter.</p>

<p>This last point advocate to avoid rejecting commits, and just break on CI on a feature branch, thus it won't block anyone else, and let the dev fix the problem early, when the change is needed and prevent this change to be forget in the flow.</p>

<p>FWIW, we did exactly this here, developers had whole access to development machines and releases in Q/A were failing because a parameter change has been forget, we did move to chef to handle the configuration of the middleware (tomcat now) so each needed change to the infrastructure has to be coded somewhere and will be reproduced in all environment.</p>

CC BY-SA 3.0
2017-11-24T23:24:48.483Z
Bonus points if the changes made by developer rely on a function from the testing framework so it works everywhere but the production environment (which I managed to do once).
Jakub Kania
CC BY-SA 3.0
2017-03-07T14:38:25.760Z
2017-03-07T14:38:25.760Z
Tensibai
<p>Sure it is.  Production is always different.  Real money.  Real load.  Real users.  Real pain.  This is why it is so important to put any significant change behind a feature flag.  Your deployment should not change anything.  Turning on a feature is the only thing that should make significant changes to your site.</p>

CC BY-SA 3.0
2017-03-07T04:57:33.437Z
I'm talking about regressions caught by the CI system - in the integration branch. Before deployment in production.
Dan Cornilescu
CC BY-SA 3.0
2017-03-07T05:01:50.997Z
Consider a CSS change that passes all tests but in production it is noticed that sales drop to $0.  Upon visual inspection it is noticed the button text is now the same color as the background and users refuse to click on a colored blob.  Automation can pass this type of problem unless you have UI regression and people actually verify the regressions.  CI may not be where this problem appears.
No Refunds No Returns
CC BY-SA 3.0
2017-03-07T05:04:33.283Z
Also covered: "all mandatory tests passed, including those covering the area in question and should have detected the regression" - the same test passed in pre-commit verification of the developer's image but failed with the CI image built after the change was committed.
Dan Cornilescu
CC BY-SA 3.0
2017-03-07T04:54:28.117Z
2017-03-07T04:54:28.117Z
No Refunds No Returns
<p>The breakage is always theoretically possible because the pre-commit verification performed by the developer is done in isolation and thus can't take into account other in-flight changes being verified in parallel. Such changes can interfere with each-other and cause regressions without actually have collissions detectable at the SCM-level.</p>

<p>A simple example of such interfering changes:</p>

<p>Let's assume the code in the latest version of a project branch includes a certain function, defined in one file and invoked in a couple of other files. Two developers working in parallel on that project are preparing to make some changes to the code.</p>

<p>Developer A reworks that function removing or adding a mandatory function argument and, of course, updates all invocations of the function in all the associated files to match the updated definition.</p>

<p>Developer B decides to add an invocation of said function in a file which didn't contain any such invocation before and is thus not touched by developer A's changes. Of course developer B is filling the argument list to match the function's definition visible in the latest label. Which is the old definition, as developer A's changes aren't yet committed.</p>

<p>Both developers correctly perform the pre-commit verifications with a pass result and proceed to commit their code changes. Since the two changesets do not touch the same files no final merge happens, which typically would be an indication of potential problems, warranting a closer look and maybe a re-execution of the pre-commit verification. Nothing whatsoever can give even a subtle hint that something may go wrong.</p>

<p>Yet the end result is catastrophic - the build is broken as the function call added by developer B doesn't match the function definition updated by developer A.</p>

CC BY-SA 3.0
2017-03-13T17:34:10.410Z
2017-03-13T17:34:10.410Z
Dan Cornilescu
<p>When you find this type of issue, you should write some new extremely quick running acceptance tests that can eke out these issues, and add them to your build verification tests that run prior to your integration tests. You should constantly be shifting left and trying to shorten the feedback loop to the developers committing changes. If you cannot find a way to do this, perhaps your architecture isn't as agile as it needs to be.</p>

<p>@Dan Cornilescu - your scenario is valid for tightly coupled architectures which is why loosely coupled architectures (microservices of versioned RESTful APIs) have emerged as the current best practice in high performing organizations. These matrix of services organizations have other complexities to overcome though.</p>

<p>Sometimes you need to refactor your entire architecture to overcome issues such as these. I believe it was both Google and eBay that have <strong>completely</strong> rearchitected their platforms 5 times (in a span of something like 10 years) due to constraints their previous architecture imposed upon them. </p>

CC BY-SA 3.0
2017-03-24T01:15:18.010Z
I think you missed the point of the question: the mandatory set of pre-commit quality verifications mentioned in the question already include the acceptance tests you're talking about. They're executed by devs prior to their commits and pass and they're executed by the CI system and they fail after both commits are in.
Dan Cornilescu
CC BY-SA 3.0
2017-03-24T01:29:37.470Z
That was my point, perhaps a completely new architecture is the appropriate response to this issue that cannot be resolved by the current architecture.
icewav
CC BY-SA 3.0
2017-03-24T01:30:26.347Z
you mean product architecture or process architecture?
Dan Cornilescu
CC BY-SA 3.0
2017-03-24T01:51:38.770Z
A new product architecture if the current process architecture can't solve the issue.
icewav
CC BY-SA 3.0
2017-03-24T01:59:32.703Z
The product architecture is irrelevant, the question applies to pretty much any product architecture, except maybe for very small, trivial sw products. See the example in my answer.
Dan Cornilescu
CC BY-SA 3.0
2017-03-24T02:11:11.423Z
Whatever solution solves the issue is what the ultimate answer is. I'm all for that solution. In my experience, rearchitecting the product has worked for those organizations doing that. If your experience differs, thats great.
icewav
CC BY-SA 3.0
2017-03-23T23:04:14.337Z
2017-03-23T23:04:14.337Z
icewav
<p>In a CI context one of the commonly-used measures of increasing the quality levels of the integration branch is a mandatory set of pre-commit quality verifications (typically including building some artifacts, performing unit tests and even some feature/integration tests).</p>

<p>Yet some regressions (build breakages, various test failures) are detected by the CI system verifications in exactly the areas which were supposed to be covered by these mandatory pre-commit verifications.</p>

<p>During analysis of these regressions an argument often heard is that the developer who committed the change identified as root-cause of the regression has successfully passed all such verifications. And often the claim is supported by hard evidence indicating that:</p>

<ul>
<li>after the final version of the change was reached it was ported to a fresh workspace based on the tip of the branch</li>
<li>the required artifacts were built from scratch (so the build was totally fine, no cache-related issues, etc.)</li>
<li>all mandatory tests passed, including those covering the area in question and should have detected the regression</li>
<li>no intermittent false-positives affected the respective verifications</li>
<li>no file merges were detected when committing the change to the branch</li>
<li>none of the files being modified was touched by any other change committed in the branch since the fresh workspace was pulled</li>
</ul>

<p>Is it really possible for a software change to cause such a regression despite correctly following all the prescribed processes and practices? How?</p>

CC BY-SA 3.0
2017-03-07T08:55:58.873Z
By pre-commit, you mean before entering the CI system (on the dev workstation) or SCM hooks checks ? I see a case on dev workstaion, but not on pre-commit hooks.
Tensibai
CC BY-SA 3.0
2017-03-07T13:55:41.240Z
I guess it depends on the exact SCM system. Yes if the pre-commit hook executions for 2 changes going into the same branch can overlap in time - as it's the case for git, for example.
Dan Cornilescu
CC BY-SA 3.0
2017-03-07T13:57:39.560Z
Sorry I was unclear, I mean does your pre-commit checks run on dev workstation or on the SCM side (central repository) ? All in all, I wonder if you're talking about checks a dev will launch himself before commit or if it is an automated testing system on 'receive' by the central server.
Tensibai
CC BY-SA 3.0
2017-03-07T14:03:49.687Z
Yes, that's where I'm heading. With the note that the regression can happen even with centralized checks - if they are allowed to overlap in time.
Dan Cornilescu
CC BY-SA 3.0
2017-03-07T14:09:19.740Z
Ok, rhetorical question then (I mean, one where you already have the answer and know what you're waiting). That doesn't leave much place for real answers at all.
Tensibai
CC BY-SA 3.0
2017-03-07T14:28:58.227Z
Well, yours would still be valuable - it covers a case I obviously didn't think about since I didn't clarified it in the question ;) Your case is also much more common, I'd suspect.
Dan Cornilescu
CC BY-SA 3.0
2017-03-24T04:05:58.597Z
I was a lead dev on the version control system developed internally at Oracle and wrote the feature that dealt with exactly those issues in automated fashion, as the pre-commits also often cannot run a full set of tests, as the nightly test suite. If you later post a question on resolving these issues, let me know. But this is exactly what causes one of the hard to solve problems of fully automated integration pipeline, which would accept, reject and rollback changes to keep the daily builds pristine.
Jiri Klouda
CC BY-SA 3.0
2017-03-24T11:18:41.390Z
@JiriKlouda see http://devops.stackexchange.com/questions/268/is-there-a-ci-tool-that-guarantees-no-regressions-in-the-branch-quality-level
Dan Cornilescu
<continuous-integration><regression>
How can successfully pre-verified changes cause regressions that should have been caught?
Dan Cornilescu
# How can successfully pre-verified changes cause regressions that should have been caught?

(Asked by: username_0 on 2017-03-07)

In a CI context one of the commonly-used measures of increasing the quality levels of the integration branch is a mandatory set of pre-commit quality verifications (typically including building some artifacts, performing unit tests and even some feature/integration tests).

Yet some regressions (build breakages, various test failures) are detected by the CI system verifications in exactly the areas which were supposed to be covered by these mandatory pre-commit verifications.

During analysis of these regressions an argument often heard is that the developer who committed the change identified as root-cause of the regression has successfully passed all such verifications. And often the claim is supported by hard evidence indicating that:

* after the final version of the change was reached it was ported to a fresh workspace based on the tip of the branch
* the required artifacts were built from scratch (so the build was totally fine, no cache-related issues, etc.)
* all mandatory tests passed, including those covering the area in question and should have detected the regression
* no intermittent false-positives affected the respective verifications
* no file merges were detected when committing the change to the branch
* none of the files being modified was touched by any other change committed in the branch since the fresh workspace was pulled

Is it really possible for a software change to cause such a regression despite correctly following all the prescribed processes and practices? How?

**Tags:** continuous-integration, regression

> **Comments:**
> **username_1**: By pre-commit, you mean before entering the CI system (on the dev workstation) or SCM hooks checks ? I see a case on dev workstaion, but not on pre-commit hooks.  
> **username_0**: I guess it depends on the exact SCM system. Yes if the pre-commit hook executions for 2 changes going into the same branch can overlap in time - as it's the case for git, for example.  
> **username_1**: Sorry I was unclear, I mean does your pre-commit checks run on dev workstation or on the SCM side (central repository) ? All in all, I wonder if you're talking about checks a dev will launch himself before commit or if it is an automated testing system on 'receive' by the central server.  
> **username_0**: Yes, that's where I'm heading. With the note that the regression can happen even with centralized checks - if they are allowed to overlap in time.  
> **username_1**: Ok, rhetorical question then (I mean, one where you already have the answer and know what you're waiting). That doesn't leave much place for real answers at all.  
> **username_0**: Well, yours would still be valuable - it covers a case I obviously didn't think about since I didn't clarified it in the question ;) Your case is also much more common, I'd suspect.  
> **username_2**: I was a lead dev on the version control system developed internally at Oracle and wrote the feature that dealt with exactly those issues in automated fashion, as the pre-commits also often cannot run a full set of tests, as the nightly test suite. If you later post a question on resolving these issues, let me know. But this is exactly what causes one of the hard to solve problems of fully automated integration pipeline, which would accept, reject and rollback changes to keep the daily builds pristine.  
> **username_0**: @JiriKlouda see http://devops.stackexchange.com/questions/268/is-there-a-ci-tool-that-guarantees-no-regressions-in-the-branch-quality-level  

---

## 4 answers

### Answer by username_1 on 2017-03-07. Score: 5

There's one possibility I can think of, if when the dev are working on their own workstation, with sometimes images baked for virtual box to run on their workstation where your infrastructure doesn't use the exact same image.

The dev will need, while developing a feature, need to add a JVM parameter or whatever change to the middleware early in its work and forget it.

Before commiting, all unit/integration tests run on its workstation works great, as the baked image is shared, it works on every develloper system.

But when going through CI, it fails because the change to the middle-ware wasn't implemented, either because the dev forgot to ask for it, or just because the team in charge of updating the base images/provisioning system didn't had the time or did forget to update the system.

That's a good thing it break in CI, because it tells early before going into production that the system won't work as expected, but sometimes it becomes a hell to find the missing parameter.

This last point advocate to avoid rejecting commits, and just break on CI on a feature branch, thus it won't block anyone else, and let the dev fix the problem early, when the change is needed and prevent this change to be forget in the flow.

FWIW, we did exactly this here, developers had whole access to development machines and releases in Q/A were failing because a parameter change has been forget, we did move to chef to handle the configuration of the middleware (tomcat now) so each needed change to the infrastructure has to be coded somewhere and will be reproduced in all environment.

> **Comments:**
> **username_3**: Bonus points if the changes made by developer rely on a function from the testing framework so it works everywhere but the production environment (which I managed to do once).  

---

### Answer by username_4 on 2017-03-07. Score: 2

Sure it is. Production is always different. Real money. Real load. Real users. Real pain. This is why it is so important to put any significant change behind a feature flag. Your deployment should not change anything. Turning on a feature is the only thing that should make significant changes to your site.

> **Comments:**
> **username_0**: I'm talking about regressions caught by the CI system - in the integration branch. Before deployment in production.  
> **username_4**: Consider a CSS change that passes all tests but in production it is noticed that sales drop to $0.  Upon visual inspection it is noticed the button text is now the same color as the background and users refuse to click on a colored blob.  Automation can pass this type of problem unless you have UI regression and people actually verify the regressions.  CI may not be where this problem appears.  
> **username_0**: Also covered: "all mandatory tests passed, including those covering the area in question and should have detected the regression" - the same test passed in pre-commit verification of the developer's image but failed with the CI image built after the change was committed.  

---

### Answer by username_0 on 2017-03-13. Score: 2

The breakage is always theoretically possible because the pre-commit verification performed by the developer is done in isolation and thus can't take into account other in-flight changes being verified in parallel. Such changes can interfere with each-other and cause regressions without actually have collissions detectable at the SCM-level.

A simple example of such interfering changes:

Let's assume the code in the latest version of a project branch includes a certain function, defined in one file and invoked in a couple of other files. Two developers working in parallel on that project are preparing to make some changes to the code.

Developer A reworks that function removing or adding a mandatory function argument and, of course, updates all invocations of the function in all the associated files to match the updated definition.

Developer B decides to add an invocation of said function in a file which didn't contain any such invocation before and is thus not touched by developer A's changes. Of course developer B is filling the argument list to match the function's definition visible in the latest label. Which is the old definition, as developer A's changes aren't yet committed.

Both developers correctly perform the pre-commit verifications with a pass result and proceed to commit their code changes. Since the two changesets do not touch the same files no final merge happens, which typically would be an indication of potential problems, warranting a closer look and maybe a re-execution of the pre-commit verification. Nothing whatsoever can give even a subtle hint that something may go wrong.

Yet the end result is catastrophic - the build is broken as the function call added by developer B doesn't match the function definition updated by developer A.


---

### Answer by username_5 on 2017-03-23. Score: 1

When you find this type of issue, you should write some new extremely quick running acceptance tests that can eke out these issues, and add them to your build verification tests that run prior to your integration tests. You should constantly be shifting left and trying to shorten the feedback loop to the developers committing changes. If you cannot find a way to do this, perhaps your architecture isn't as agile as it needs to be.

@username_0 - your scenario is valid for tightly coupled architectures which is why loosely coupled architectures (microservices of versioned RESTful APIs) have emerged as the current best practice in high performing organizations. These matrix of services organizations have other complexities to overcome though.

Sometimes you need to refactor your entire architecture to overcome issues such as these. I believe it was both Google and eBay that have **completely** rearchitected their platforms 5 times (in a span of something like 10 years) due to constraints their previous architecture imposed upon them.

> **Comments:**
> **username_0**: I think you missed the point of the question: the mandatory set of pre-commit quality verifications mentioned in the question already include the acceptance tests you're talking about. They're executed by devs prior to their commits and pass and they're executed by the CI system and they fail after both commits are in.  
> **username_5**: That was my point, perhaps a completely new architecture is the appropriate response to this issue that cannot be resolved by the current architecture.  
> **username_0**: you mean product architecture or process architecture?  
> **username_5**: A new product architecture if the current process architecture can't solve the issue.  
> **username_0**: The product architecture is irrelevant, the question applies to pretty much any product architecture, except maybe for very small, trivial sw products. See the example in my answer.  
> **username_5**: Whatever solution solves the issue is what the ultimate answer is. I'm all for that solution. In my experience, rearchitecting the product has worked for those organizations doing that. If your experience differs, thats great.  

---

