<p>I am not sure about other approaches, but the one that I use is a modification of #3.</p>

<p>There's usually a classification of releases (for example think of Ubuntu LTS and regular release).</p>

<ol>
<li>For the regular release, I would just release a new version and add a new tag.</li>
<li>For LTS/Stable releases I would delete a tag pointing to a faulty release or change a commit this tag points to</li>
</ol>

CC BY-SA 4.0
2020-04-26T14:24:23.317Z
1. is understandable and seems like it resolves the problem without introducing complexity. As for 2: I heard from some senior people I should never remove tags, because changing/deleting history usually does more harm than good.
pjboro
CC BY-SA 4.0
2020-04-26T17:28:30.157Z
@pjboro I would not consider tag a history - it is just a pointer to a commit. You may also consider changing the commit a tag points to
Alexander Pushkarev
CC BY-SA 4.0
2020-04-26T13:38:33.540Z
2020-04-26T17:28:53.730Z
2020-04-26T17:28:53.730Z
Alexander Pushkarev
<p>I'm building a product to solve this problem among others - <a href="https://relizahub.com" rel="nofollow noreferrer">https://relizahub.com</a> (note it's pretty early stage at the moment).</p>

<p>Idea goes as following - for all your releases you stream their details to Reliza Hub. You would then have a client that would pick up releases you need based on combination of tags, approvals, etc. So it is somewhat similar to your #1 approach. For broken releases you either do not approve them or mark them as 'REJECTED' and this would stop them from propagating in your pipeline.</p>

CC BY-SA 4.0
2020-04-26T22:14:29.983Z
GitLab seems like enough right now, but... I'll read on the docs on relizahub :)
pjboro
CC BY-SA 4.0
2020-04-26T22:42:38.957Z
@pjboro Sounds good :) Reliza Hub is at a very early stage right now and I'm very open to talk personally to solve specific use cases - feel free to connect via my linkedin: https://www.linkedin.com/in/pshukhman/
taleodor
CC BY-SA 4.0
2020-04-26T15:53:43.613Z
2020-04-26T15:53:43.613Z
taleodor
<p>I think that it might help to distinguish between a "release" and a "release candidate". If you really have shipped a version of software to other people and it is 'totally broken' then something went really very badly wrong. To avoid this opensource projects have milestone release and snapshot release (aka "nighly releases" or "freshly releases"). These try to flush out critical failures through pre-release public testing. Once the risk of shipping something totally broken has passed then the software is tagged as the final release. After that, if there are bugs, and there always are bugs, then it is a matter of issuing advisories and warning folks about bad old versions within release notes.  </p>

<p>For example, spring.io <a href="https://search.maven.org/artifact/org.springframework/spring-core" rel="nofollow noreferrer">releases</a> are versioned something like "5.2.5.RELEASE". We might expect that they have a "5.2" branch and that they had many release candidate builds on that branch. In fact they have actually have two special repos for milestone releases and snapshot releases as <a href="https://www.baeldung.com/spring-maven-repository" rel="nofollow noreferrer">described here</a>. These approximate to beta releases and alpha releases. </p>

<p>If you are not releasing software to other people to run, but are running the software as a service yourself, then your option 3 may work well: </p>

<blockquote>
  <p>(3) Do no save information about release problems: just deploy a new stable, patched version to production and never look back.</p>
</blockquote>

<p>This approach can work well if you are a building a web-app or mobile backend based product. The internal software at large firms like global banks will typically use this approach. In fact, I have worked on many systems at both banks and startups over two decades and we only ever used this approach in practice. </p>

<p>I should mention that your Git server might have a "latest release feature". Both GitHub and GitHub Enterprise have this. Even if you release via a build pipeline to some artefact repository it can still be helpful to make a "release" within the Git server so that an API call to something like <code>GET /repos/:owner/:repo/releases/latest</code> API will return details of the latest version including the git tag that should match the release version. Your deployment piplien can then pull the corresponding prebuilt code from the artefact server to deploy. </p>

CC BY-SA 4.0
2020-05-09T13:11:16.047Z
2020-05-09T13:11:16.047Z
simbo1905
<p>I spent considerable time googling around, but found nothing that would answer my question, so here it is.</p>

<p>If you do a canary release, and you learn the the release is somehow broken: how to you denote that a certain tag is broken and should not be deployed anywhere else, but removed from your productions ASAP?</p>

<p>I had several ideas:</p>

<ol>
<li><p>Using GitLab, I could add something like <code>**broken**</code> to tag release note. Deploy jobs would call GitLab API to ensure that the tag I want to deploy is not annotated as broken.</p></li>
<li><p>I could have a text file in repo, where broken tags would be listed. In this case deploy job checks if the tag you want to deploy is not listed in the file.</p></li>
<li><p>Do no save information about release problems: just deploy a new stable, patched version to production and never look back.</p></li>
</ol>

<p>How do/would you handle this? Any tips or hyperlinks are welcome.</p>

<release-management>
How to keep track of broken releases?
pjboro
# How to keep track of broken releases?

(Asked by: username_0 on 2020-04-25)

I spent considerable time googling around, but found nothing that would answer my question, so here it is.

If you do a canary release, and you learn the the release is somehow broken: how to you denote that a certain tag is broken and should not be deployed anywhere else, but removed from your productions ASAP?

I had several ideas:

1. Using GitLab, I could add something like `**broken**` to tag release note. Deploy jobs would call GitLab API to ensure that the tag I want to deploy is not annotated as broken.
2. I could have a text file in repo, where broken tags would be listed. In this case deploy job checks if the tag you want to deploy is not listed in the file.
3. Do no save information about release problems: just deploy a new stable, patched version to production and never look back.

How do/would you handle this? Any tips or hyperlinks are welcome.

**Tags:** release-management


---

## 3 answers

### Answer by username_1 on 2020-04-26. Score: 2

I am not sure about other approaches, but the one that I use is a modification of #3.

There's usually a classification of releases (for example think of Ubuntu LTS and regular release).

1. For the regular release, I would just release a new version and add a new tag.
2. For LTS/Stable releases I would delete a tag pointing to a faulty release or change a commit this tag points to

> **Comments:**
> **username_0**: 1. is understandable and seems like it resolves the problem without introducing complexity. As for 2: I heard from some senior people I should never remove tags, because changing/deleting history usually does more harm than good.  
> **username_1**: @username_0 I would not consider tag a history - it is just a pointer to a commit. You may also consider changing the commit a tag points to  

---

### Answer by username_2 on 2020-04-26. Score: 2

I'm building a product to solve this problem among others - <https://relizahub.com> (note it's pretty early stage at the moment).

Idea goes as following - for all your releases you stream their details to Reliza Hub. You would then have a client that would pick up releases you need based on combination of tags, approvals, etc. So it is somewhat similar to your #1 approach. For broken releases you either do not approve them or mark them as 'REJECTED' and this would stop them from propagating in your pipeline.

> **Comments:**
> **username_0**: GitLab seems like enough right now, but... I'll read on the docs on relizahub :)  
> **username_2**: @username_0 Sounds good :) Reliza Hub is at a very early stage right now and I'm very open to talk personally to solve specific use cases - feel free to connect via my linkedin: https://www.linkedin.com/in/pshukhman/  

---

### Answer by username_3 on 2020-05-09. Score: 1

I think that it might help to distinguish between a "release" and a "release candidate". If you really have shipped a version of software to other people and it is 'totally broken' then something went really very badly wrong. To avoid this opensource projects have milestone release and snapshot release (aka "nighly releases" or "freshly releases"). These try to flush out critical failures through pre-release public testing. Once the risk of shipping something totally broken has passed then the software is tagged as the final release. After that, if there are bugs, and there always are bugs, then it is a matter of issuing advisories and warning folks about bad old versions within release notes.

For example, spring.io [releases](https://search.maven.org/artifact/org.springframework/spring-core) are versioned something like "5.2.5.RELEASE". We might expect that they have a "5.2" branch and that they had many release candidate builds on that branch. In fact they have actually have two special repos for milestone releases and snapshot releases as [described here](https://www.baeldung.com/spring-maven-repository). These approximate to beta releases and alpha releases.

If you are not releasing software to other people to run, but are running the software as a service yourself, then your option 3 may work well:

> (3) Do no save information about release problems: just deploy a new stable, patched version to production and never look back.

This approach can work well if you are a building a web-app or mobile backend based product. The internal software at large firms like global banks will typically use this approach. In fact, I have worked on many systems at both banks and startups over two decades and we only ever used this approach in practice.

I should mention that your Git server might have a "latest release feature". Both GitHub and GitHub Enterprise have this. Even if you release via a build pipeline to some artefact repository it can still be helpful to make a "release" within the Git server so that an API call to something like `GET /repos/:owner/:repo/releases/latest` API will return details of the latest version including the git tag that should match the release version. Your deployment piplien can then pull the corresponding prebuilt code from the artefact server to deploy.


---

