<p>As a comment mentions, your question is pretty broad.</p>
<p>In my experience the main problem is the fact that you have multiple different teams working on all the services. So that is the main issue my answer tries to attack:</p>
<h2>Central definition of deployment versions</h2>
<p>The first approach would be to leave the teams disjoint, including their separate clusters, and sticking with your approach to have dual clusters for each team (note that having separate namespaces would be a little more lightweight, but may make everything a bit more complicated depending on how you have everything set up, especially concerning config and secret management).</p>
<p>But you would then chose a central approach for <em>deploying</em> everything. You could have a central ArgoCD repository (a single repo for all your clusters/teams), and each of the separate clusters could reference the same ArgoCD repo. You could use one ApplicationSet per service (or per cluster, or per team, depending on how big everything is and how much individual control you want to have); and you would use an appropriate directory structure within that repo to keep everything well separated.</p>
<p>This repo would be managed by a <em>single</em> team in the beginning, at least, and possibly forever. This single team would be responsible for discussing changes, backward compatibility problems and so on and forth between the other teams.</p>
<p>This would mean that the individual teams would need to be very clear with their communication (i.e., write proper Changelogs, have proper semantic versioning and so on and forth), which is good practice anyways. And there would be of course some communication overhead, and they would give up a little bit of autonomy regarding their deployments, but depending on your situation that may all be acceptable.</p>
<p>Eventually, when everything has settled in, you might chose to reduce the role of the central team and try to distribute the work of updating the central repo over the teams again, if you trust them not to break down in chaos.</p>
<h2>Hardcore backward compatibility</h2>
<p>The second approach would be to let everybody deploy their own stuff, but work with very strict guidelines that breaking changes can <em>never</em> be done on existing APIs. Everytime an API changes in any way that means it would not be backward compatible, you would enforce the API name to change (e.g., increase &quot;.../v1/...&quot; in the REST-API URL to &quot;.../v2/...&quot; and so on and forth). Every service would be forced, in this scenario, to support older versions indefinitely. You would then maybe have regularly (quarterly, no matter) meetings where you could officially phase out old versions, but  only when there are no users of said versions anymore.</p>
<p>You could achieve this by either keeping separate implementations in the code itself alive, or by keeping older images (and deployments of them) around and solving the issue on the k8s ingress level.</p>
<h2>N.B. on config management</h2>
<p>Be aware that having this kind of fluent upgrades (blue/green) can be made much worse if you do naive management of ConfigMaps. At the very least, you <em>must</em> have ConfigMaps in git repos as well (not Secrets, of course). Optimally, you use something like kustomize's <a href="https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/" rel="nofollow noreferrer">content-hash-based ConfigMap names</a>. This means you have no issues keeping multiple versions of ConfigMaps around along with the deployments needing them, and if you switch back and forth between your blue/green deployments, each of them keeps their individual versions of the same ConfigMap.</p>
<h2>N.B. on circular dependencies</h2>
<p>Just don't do it. You don't want to have two services which depend on each other, or several services which form a circle and can only be deployed in one feel swoop - especially not crossing cluster boundaries. If you find something like this, do yourself a favor and break the circle before you do anything else.</p>

CC BY-SA 4.0
2024-05-28T09:41:58.853Z
About the first option - My approach was not a cluster per team. We have "rings" - one for dev, qa, production and such and each ring can have several clusters - a single cluster for a region with several regions per ring. So I thought of having an additional cluster per region. A single repo for everyone won't happen.. There's a repo per service and a team owning that service and repo, each service owned by a different team.
CodeMonkey
CC BY-SA 4.0
2024-05-28T09:43:39.857Z
Second option - This is already happening but it doesn't really let me the option of testing new features or changes to existing features when the API still remains the same
CodeMonkey
CC BY-SA 4.0
2024-05-28T10:13:34.440Z
@CodeMonkey, in the second option, all new features (which break older ones) and especially all changes to existing features would be placed in a new API version "namespace", i.e. assuming you are using regular conventions any URLs of the form /prefix/api/v1/something would simply increase the "v1" API version number. And obviously extra effort to keep the old stuff alive, maybe with some kind of internal compatibility layer... Does that help?
AnoE
CC BY-SA 4.0
2024-05-28T10:30:57.323Z
Then you mean that every user will face the public "main" api version of v1 and every new feature and test will be done on a new version which later be merged to v1? That sounds like a lot of work always adding a new endpoint. Also, there are grpc services which are accessible via an SDK nuget and I'm not sure if you can have different nuget version for different versions? It also doesn't really sounds like Green Blue deployments..
CodeMonkey
CC BY-SA 4.0
2024-05-28T11:16:38.177Z
No, don't merge down, but keep the different versions alive in parallel either in code or via parallel deployments of old and new version(s). Whether this is much work or not depends wholely on your application and the kind of changes - if it is too much, then don't do it. It is just one way one could approach this. This means that the *consumer* of an API decides when a change effectively takes place, and reduces complexity (new features can be deployed anytime without much coordination).
AnoE
CC BY-SA 4.0
2024-05-28T11:25:39.937Z
At first I was against it because I thought it would be annoying for users to keep switching the url for a new endpoint, but if we let them choose when they want to move it might be possible... And what about having 2 different api versions using different nuget versions? You can't add several versions of the same package to the project
CodeMonkey
CC BY-SA 4.0
2024-05-28T08:24:00.020Z
2024-05-28T08:24:00.020Z
AnoE
<p>I am working in Azure in an AKS cluster environment. The environment has several clusters separated from each other. Each cluster is hosting several containers of services, each of them is developed and maintain by a different team.</p>
<p>I've been reading about Green/Blue deployments about testing a change in one cluster and then rerouting traffic to that cluster once everything is ok. For that I will have an additional cluster for each existing cluster.</p>
<p>My question is about having several teams deploying changes, each to its own service. If testing is made on a change of a single team, how do you suggest handling multiple teams all wanting to make changes at the same time? Also, how do you make sure a certain change testing doesn't starve other changes from being deployed and tested? These questions make me wonder if I might going in the wrong direction adding another cluster for each existing cluster?</p>

CC BY-SA 4.0
2024-05-28T01:22:42.437Z
The question is pretty broad, I'm building a product - Reliza Hub - that solves most of these pains. However, it usually requires some consulting approach to implement. Generally, I believe you're looking at our concept of Bundling - https://docs.relizahub.com/bundling/ - deploying various bundle versions onto different clusters or different namespaces. This way via proper versioning changes can be tested in isolation and without stomping on each other.

Generally, I suggest you find me at devopscommunity.org Discord and we can discuss this on a call.
taleodor
<azure-devops><deployment><continuous-deployment><bluegreen>
Green/Blue Deployments with multiple development teams
CodeMonkey
# Green/Blue Deployments with multiple development teams

(Asked by: username_0 on 2024-05-27)

I am working in Azure in an AKS cluster environment. The environment has several clusters separated from each other. Each cluster is hosting several containers of services, each of them is developed and maintain by a different team.

I've been reading about Green/Blue deployments about testing a change in one cluster and then rerouting traffic to that cluster once everything is ok. For that I will have an additional cluster for each existing cluster.

My question is about having several teams deploying changes, each to its own service. If testing is made on a change of a single team, how do you suggest handling multiple teams all wanting to make changes at the same time? Also, how do you make sure a certain change testing doesn't starve other changes from being deployed and tested? These questions make me wonder if I might going in the wrong direction adding another cluster for each existing cluster?

**Tags:** azure-devops, deployment, continuous-deployment, bluegreen

> **Comments:**
> **username_1**: The question is pretty broad, I'm building a product - Reliza Hub - that solves most of these pains. However, it usually requires some consulting approach to implement. Generally, I believe you're looking at our concept of Bundling - https://docs.relizahub.com/bundling/ - deploying various bundle versions onto different clusters or different namespaces. This way via proper versioning changes can be tested in isolation and without stomping on each other.  Generally, I suggest you find me at devopscommunity.org Discord and we can discuss this on a call.  

---

## 1 answers

### Answer by username_2 on 2024-05-28. Score: 0

As a comment mentions, your question is pretty broad.

In my experience the main problem is the fact that you have multiple different teams working on all the services. So that is the main issue my answer tries to attack:

## Central definition of deployment versions

The first approach would be to leave the teams disjoint, including their separate clusters, and sticking with your approach to have dual clusters for each team (note that having separate namespaces would be a little more lightweight, but may make everything a bit more complicated depending on how you have everything set up, especially concerning config and secret management).

But you would then chose a central approach for *deploying* everything. You could have a central ArgoCD repository (a single repo for all your clusters/teams), and each of the separate clusters could reference the same ArgoCD repo. You could use one ApplicationSet per service (or per cluster, or per team, depending on how big everything is and how much individual control you want to have); and you would use an appropriate directory structure within that repo to keep everything well separated.

This repo would be managed by a *single* team in the beginning, at least, and possibly forever. This single team would be responsible for discussing changes, backward compatibility problems and so on and forth between the other teams.

This would mean that the individual teams would need to be very clear with their communication (i.e., write proper Changelogs, have proper semantic versioning and so on and forth), which is good practice anyways. And there would be of course some communication overhead, and they would give up a little bit of autonomy regarding their deployments, but depending on your situation that may all be acceptable.

Eventually, when everything has settled in, you might chose to reduce the role of the central team and try to distribute the work of updating the central repo over the teams again, if you trust them not to break down in chaos.

## Hardcore backward compatibility

The second approach would be to let everybody deploy their own stuff, but work with very strict guidelines that breaking changes can *never* be done on existing APIs. Everytime an API changes in any way that means it would not be backward compatible, you would enforce the API name to change (e.g., increase ".../v1/..." in the REST-API URL to ".../v2/..." and so on and forth). Every service would be forced, in this scenario, to support older versions indefinitely. You would then maybe have regularly (quarterly, no matter) meetings where you could officially phase out old versions, but only when there are no users of said versions anymore.

You could achieve this by either keeping separate implementations in the code itself alive, or by keeping older images (and deployments of them) around and solving the issue on the k8s ingress level.

## N.B. on config management

Be aware that having this kind of fluent upgrades (blue/green) can be made much worse if you do naive management of ConfigMaps. At the very least, you *must* have ConfigMaps in git repos as well (not Secrets, of course). Optimally, you use something like kustomize's [content-hash-based ConfigMap names](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/). This means you have no issues keeping multiple versions of ConfigMaps around along with the deployments needing them, and if you switch back and forth between your blue/green deployments, each of them keeps their individual versions of the same ConfigMap.

## N.B. on circular dependencies

Just don't do it. You don't want to have two services which depend on each other, or several services which form a circle and can only be deployed in one feel swoop - especially not crossing cluster boundaries. If you find something like this, do yourself a favor and break the circle before you do anything else.

> **Comments:**
> **username_0**: About the first option - My approach was not a cluster per team. We have "rings" - one for dev, qa, production and such and each ring can have several clusters - a single cluster for a region with several regions per ring. So I thought of having an additional cluster per region. A single repo for everyone won't happen.. There's a repo per service and a team owning that service and repo, each service owned by a different team.  
> **username_0**: Second option - This is already happening but it doesn't really let me the option of testing new features or changes to existing features when the API still remains the same  
> **username_2**: @username_0, in the second option, all new features (which break older ones) and especially all changes to existing features would be placed in a new API version "namespace", i.e. assuming you are using regular conventions any URLs of the form /prefix/api/v1/something would simply increase the "v1" API version number. And obviously extra effort to keep the old stuff alive, maybe with some kind of internal compatibility layer... Does that help?  
> **username_0**: Then you mean that every user will face the public "main" api version of v1 and every new feature and test will be done on a new version which later be merged to v1? That sounds like a lot of work always adding a new endpoint. Also, there are grpc services which are accessible via an SDK nuget and I'm not sure if you can have different nuget version for different versions? It also doesn't really sounds like Green Blue deployments..  
> **username_2**: No, don't merge down, but keep the different versions alive in parallel either in code or via parallel deployments of old and new version(s). Whether this is much work or not depends wholely on your application and the kind of changes - if it is too much, then don't do it. It is just one way one could approach this. This means that the *consumer* of an API decides when a change effectively takes place, and reduces complexity (new features can be deployed anytime without much coordination).  
> **username_0**: At first I was against it because I thought it would be annoying for users to keep switching the url for a new endpoint, but if we let them choose when they want to move it might be possible... And what about having 2 different api versions using different nuget versions? You can't add several versions of the same package to the project  

---

