<blockquote>
<p>Does this procedure make sense as a solution to my use case?</p>
</blockquote>
<p>vscode devcontainers are one of the greatest inventions in the DevOps space and I can't believe more people are not talking about them. I think having the local dev environment and the pipeline in sync makes a lot of sense. Though I think the procedure you outlined above could be simplified.</p>
<blockquote>
<p>I can't imagine being the first to realize this use case. Is there an
existing mechanism (as part of the docker utilities, as part of the
Azure DevOps Pipelines framework or from something completely
different) that fulfills my needs?</p>
</blockquote>
<p>I'm currently doing this in some of my <a href="https://github.com/DontShaveTheYak/terraform-module-template/blob/main/.github/workflows/test.yml" rel="nofollow noreferrer">side projects</a>. It is also supported on <a href="https://github.com/devcontainers/ci/blob/main/docs/azure-devops-task.md" rel="nofollow noreferrer">azure</a>.</p>

CC BY-SA 4.0
2022-08-02T06:58:42.693Z
As mentioned above my questions are about using the `Dockerfile` hash to tag the docker image to get a commit-specific build environment without building the docker image on each and every commit.
Roland Sarrazin
CC BY-SA 4.0
2022-08-02T05:25:56.303Z
2022-08-02T05:25:56.303Z
Levi
<p>Docker has this capability built-in and you can use it in a variety of ways.</p>
<p>One way would be to keep a &quot;cache&quot; tag for every build. This is good if your repo normally doesn't have a lot of parallel work being done.</p>
<pre><code>docker pull ${MY_IMAGE}:cache || true
docker build --cache-from ${MY_IMAGE}:cache -t ${MY_IMAGE}:${VERSION} -t ${MY_IMAGE}:cache .
docker push ${MY_IMAGE}:${VERSION}
docker push ${MY_IMAGE}:cache
</code></pre>
<p>If your repo does have a lot of parallel work being done, don't create a cache tag, instead use the latest version.</p>
<pre><code>docker pull ${MY_IMAGE}:latest || true
docker build --cache-from ${MY_IMAGE}:latest -t ${MY_IMAGE}:${VERSION} .
docker push ${MY_IMAGE}:${VERSION}
</code></pre>
<p>This is basically the following the process you outlined in your question but instead, we are relying on docker to do the work for us. The first step is to pull the image layers down locally. This is faster (usually) than building a new image and if your CI/CD is setup correctly should already be present for the use of caching.</p>
<p>Then the <code>--cache-from</code> flag tells Docker to use that image as cache. Which will prevent Docker from building any layers that haven't changed.</p>

CC BY-SA 4.0
2022-08-04T09:25:10.677Z
You would then put the `Dockerfile` hash into `$VERSION`, correct? I fail at understanding how old builds would be able to use the old image (i.e. the very same image that used to be the current one at that time) and not the current latest one.
Roland Sarrazin
CC BY-SA 4.0
2022-08-04T18:14:11.643Z
I guess it's a bad example because in my pipeline I'm building and deploying an application using semantic versioning. 

But it doesn't matter what you label the new image as, you can even remove the version label. 

The magic is in --cache-from. Cache from makes sure that docker doesn't build image layers that didnt chage.
Levi
CC BY-SA 4.0
2022-08-05T09:42:31.540Z
https://docs.docker.com/engine/reference/commandline/build/#specifying-external-cache-sources seems to require `--build-arg BUILDKIT_INLINE_CACHE=1` to be set when building the initial image.
Roland Sarrazin
CC BY-SA 4.0
2022-08-05T14:51:22.653Z
Why dont you just try it @RolandSarrazin ? This command works for me and I am not using build args and passing BUILDKIT_INLINE_CACHE in my image. I think that is required for external cache sources and not for local caches.
Levi
CC BY-SA 4.0
2022-08-02T13:12:11.770Z
2022-08-02T13:12:11.770Z
Levi
<p>Given a repository containing a <code>Dockerfile</code> that defines the build environment used by the CI pipeline as well as by the developer (e.g. as a <a href="https://code.visualstudio.com/docs/remote/create-dev-container" rel="nofollow noreferrer">Visual Studio Code devcontainer</a>), the CI pipeline shall fulfill the following requirements:</p>
<ol>
<li>The CI pipeline shall build the repository's contents in the context of a docker image built from the very <code>Dockerfile</code> present in the working branch (or the result of its merge to the target branch where applicable in the context of a pull request).</li>
</ol>
<ul>
<li>Rationale: Builds shall be deterministic so that old repository versions shall be built with their original docker image, not with the newest one.</li>
<li>Rationale: Changes to the <code>Dockerfile</code> shall be considered automatically without the need for manual user intervention (i.e. local docker build followed by a manual push).</li>
</ul>
<ol start="2">
<li>The CI pipeline shall reuse an already built docker image if the applicable <code>Dockerfile</code> has already been &quot;baked&quot; into such an image pushed to the registry.</li>
</ol>
<ul>
<li>Rationale: Building a docker image on each commit is resource-consuming (time, energy).</li>
</ul>
<p>I couldn't find an existing best practice to implement that out of the box (my CI environment being Azure DevOps Pipelines if it matters) so I came with the following concept:</p>
<ul>
<li>Calculate the <code>Dockerfile</code>'s hash.</li>
<li>Load <code>docker_image_name:$hash</code> from the docker registry.</li>
<li>If the load fails, build <code>docker_image_name:$hash</code> from the <code>Dockerfile</code> and push it to the docker registry.</li>
<li>Use <code>docker_image_name:$hash</code> (from the registry / from the local cache) to run the CI pipeline (using Azure's <a href="https://docs.microsoft.com/en-us/azure/devops/pipelines/process/container-phases?view=azure-devops#endpoints" rel="nofollow noreferrer">container jobs</a> in my case).</li>
</ul>
<p>Questions:</p>
<ol>
<li>Does this procedure make sense as a solution to my use case?</li>
<li>I can't imagine being the first to realize this use case. Is there an existing mechanism (as part of the docker utilities, as part of the Azure DevOps Pipelines framework or from something completely different) that fulfills my needs?</li>
</ol>

CC BY-SA 4.0
2022-08-02T05:28:46.440Z
I wasn't exactly sure if you was asking how to use a vscode dev container in a azure pipeline or if you was asking how to replicate that dev container functionality from scratch.
Levi
CC BY-SA 4.0
2022-08-02T06:53:09.913Z
@Levi: My mentioning vscode's devcontainers might have been misleading since it was just part of the context, but not part of the actual question. The question was specifically about how to use the very docker image specified by the currently valid `Dockerfile` without building the image on each and every commit.
Roland Sarrazin
CC BY-SA 4.0
2022-08-02T13:13:43.387Z
I think the question boils down to "How do I avoid rebuilding my container image in CICD" which probably has been asked and answered before. I went ahead and gave a new answer though.
Levi
<docker><continuous-integration><dockerfile><azure-pipelines>
How do I run a CI build in a docker image matching the current `Dockerfile` while being resource-aware?
Roland Sarrazin
# How do I run a CI build in a docker image matching the current `Dockerfile` while being resource-aware?

(Asked by: username_0 on 2022-08-01)

Given a repository containing a `Dockerfile` that defines the build environment used by the CI pipeline as well as by the developer (e.g. as a [Visual Studio Code devcontainer](https://code.visualstudio.com/docs/remote/create-dev-container)), the CI pipeline shall fulfill the following requirements:

1. The CI pipeline shall build the repository's contents in the context of a docker image built from the very `Dockerfile` present in the working branch (or the result of its merge to the target branch where applicable in the context of a pull request).

* Rationale: Builds shall be deterministic so that old repository versions shall be built with their original docker image, not with the newest one.
* Rationale: Changes to the `Dockerfile` shall be considered automatically without the need for manual user intervention (i.e. local docker build followed by a manual push).

2. The CI pipeline shall reuse an already built docker image if the applicable `Dockerfile` has already been "baked" into such an image pushed to the registry.

* Rationale: Building a docker image on each commit is resource-consuming (time, energy).

I couldn't find an existing best practice to implement that out of the box (my CI environment being Azure DevOps Pipelines if it matters) so I came with the following concept:

* Calculate the `Dockerfile`'s hash.
* Load `docker_image_name:$hash` from the docker registry.
* If the load fails, build `docker_image_name:$hash` from the `Dockerfile` and push it to the docker registry.
* Use `docker_image_name:$hash` (from the registry / from the local cache) to run the CI pipeline (using Azure's [container jobs](https://docs.microsoft.com/en-us/azure/devops/pipelines/process/container-phases?view=azure-devops#endpoints) in my case).

Questions:

1. Does this procedure make sense as a solution to my use case?
2. I can't imagine being the first to realize this use case. Is there an existing mechanism (as part of the docker utilities, as part of the Azure DevOps Pipelines framework or from something completely different) that fulfills my needs?

**Tags:** docker, continuous-integration, dockerfile, azure-pipelines

> **Comments:**
> **username_1**: I wasn't exactly sure if you was asking how to use a vscode dev container in a azure pipeline or if you was asking how to replicate that dev container functionality from scratch.  
> **username_0**: @username_1: My mentioning vscode's devcontainers might have been misleading since it was just part of the context, but not part of the actual question. The question was specifically about how to use the very docker image specified by the currently valid `Dockerfile` without building the image on each and every commit.  
> **username_1**: I think the question boils down to "How do I avoid rebuilding my container image in CICD" which probably has been asked and answered before. I went ahead and gave a new answer though.  

---

## 2 answers

### Answer by username_1 on 2022-08-02. Score: 0

> Does this procedure make sense as a solution to my use case?

vscode devcontainers are one of the greatest inventions in the DevOps space and I can't believe more people are not talking about them. I think having the local dev environment and the pipeline in sync makes a lot of sense. Though I think the procedure you outlined above could be simplified.

> I can't imagine being the first to realize this use case. Is there an
> existing mechanism (as part of the docker utilities, as part of the
> Azure DevOps Pipelines framework or from something completely
> different) that fulfills my needs?

I'm currently doing this in some of my [side projects](https://github.com/DontShaveTheYak/terraform-module-template/blob/main/.github/workflows/test.yml). It is also supported on [azure](https://github.com/devcontainers/ci/blob/main/docs/azure-devops-task.md).

> **Comments:**
> **username_0**: As mentioned above my questions are about using the `Dockerfile` hash to tag the docker image to get a commit-specific build environment without building the docker image on each and every commit.  

---

### Answer by username_1 on 2022-08-02. Score: 0

Docker has this capability built-in and you can use it in a variety of ways.
One way would be to keep a "cache" tag for every build. This is good if your repo normally doesn't have a lot of parallel work being done.
docker pull ${MY_IMAGE}:cache || true
docker build --cache-from ${MY_IMAGE}:cache -t ${MY_IMAGE}:${VERSION} -t ${MY_IMAGE}:cache .
docker push ${MY_IMAGE}:${VERSION}
docker push ${MY_IMAGE}:cache

If your repo does have a lot of parallel work being done, don't create a cache tag, instead use the latest version.
docker pull ${MY_IMAGE}:latest || true
docker build --cache-from ${MY_IMAGE}:latest -t ${MY_IMAGE}:${VERSION} .
docker push ${MY_IMAGE}:${VERSION}

This is basically the following the process you outlined in your question but instead, we are relying on docker to do the work for us. The first step is to pull the image layers down locally. This is faster (usually) than building a new image and if your CI/CD is setup correctly should already be present for the use of caching.
Then the --cache-from flag tells Docker to use that image as cache. Which will prevent Docker from building any layers that haven't changed.

> **Comments:**
> **username_0**: You would then put the `Dockerfile` hash into `$VERSION`, correct? I fail at understanding how old builds would be able to use the old image (i.e. the very same image that used to be the current one at that time) and not the current latest one.  
> **username_1**: I guess it's a bad example because in my pipeline I'm building and deploying an application using semantic versioning.   But it doesn't matter what you label the new image as, you can even remove the version label.   The magic is in --cache-from. Cache from makes sure that docker doesn't build image layers that didnt chage.  
> **username_0**: https://docs.docker.com/engine/reference/commandline/build/#specifying-external-cache-sources seems to require `--build-arg BUILDKIT_INLINE_CACHE=1` to be set when building the initial image.  
> **username_1**: Why dont you just try it @RolandSarrazin ? This command works for me and I am not using build args and passing BUILDKIT_INLINE_CACHE in my image. I think that is required for external cache sources and not for local caches.  

---

