<p>Caching of layers with buildkit in an external registry requires an extra step or two depending on how you want to cache your layers. The easy option is to include a build arg that enables the inline cache:</p>

<pre><code>docker build --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from ${LATEST} -t ${LATEST} -t ${IMAGE_COMMIT_TAG} .
</code></pre>

<p>Note that the inline cache only caches layers for the target stage that was pushed in the image, so other states in a multi-stage build would need to be built and cached separately, or rebuilt without caching, neither of which is ideal.</p>

<p>You can also cache to a local file, or push the cache to a different registry image rather than inline with the image you pushed. Unfortunately the standard <code>docker build</code> CLI doesn't have access to all the buildkit flags to enable this. Instead, you can install buildkit directly or use buildx which is a CLI plugin for managing buildkit. You can install buildx separately but in current releases it's available with experimental CLI options that can be enabled with <code>export DOCKER_CLI_EXPERIMENTAL=enabled</code>. I believe you'd need to create a container based builder for all of the buildkit options, which can be done with:</p>

<pre><code>docker buildx create --use --driver docker-container --name local ${DOCKER_HOST:-unix:///var/run/docker.sock}
</code></pre>

<p>Then you should be able to run something like:</p>

<pre><code>docker buildx build --cache-from ${IMG_CACHE} --cache-to ${IMG_CACHE} -t ${LATEST} -t ${IMAGE_COMMIT_TAG} .
</code></pre>

<p>More documentation on the buildkit caching is available at: <a href="https://github.com/moby/buildkit#export-cache" rel="nofollow noreferrer">https://github.com/moby/buildkit#export-cache</a></p>

<p>More details on buildx is available at: <a href="https://github.com/docker/buildx" rel="nofollow noreferrer">https://github.com/docker/buildx</a></p>

CC BY-SA 4.0
2020-04-01T10:58:33.327Z
HI, @BMitch!
Sorry for such a late response (I was offline up to now) and thank you for your answer.

I know about `buildx`, but it's not applicable, as I don't have any Docker stuff persistent. I have immutable GL-runner :( and that's why I have to use Docker as service.
They same reason is about using external cache.
acd
CC BY-SA 4.0
2020-04-01T11:02:25.693Z
BTW, when I run the build-stage without `DinD` (on different GL-runner) and any external cache, Docker uses cache from the built images not its own build-cache.
acd
CC BY-SA 4.0
2020-04-01T11:51:26.267Z
@acd an immutable runner would not be able to build an image. Perhaps you mean an ephemeral runner, which should not prevent you from running docker commands.
BMitch
CC BY-SA 4.0
2020-04-01T16:50:33.823Z
No-no. "Immutable runner" means persistent instance, which can't be changed, that is to setup `buildkitd`, `buildx`, as a Docker extension, etc.. And, Docker is ephemeral, which is Docker-in-Docker, used in `services` of .gitlab-ci.
acd
CC BY-SA 4.0
2020-04-01T18:02:41.547Z
@acd I'm not finding much in my search for gitlab immutable runner, perhaps you're referring to a shared runner? Immutable would mean no write access to /var/lib/docker which would mean all image builds would fail because of a read-only filesystem. It's also not clear from the comments that you attempted all three of the options suggested above. Output of the errors would help.
BMitch
CC BY-SA 4.0
2020-04-01T19:26:03.233Z
Sorry for confusion. I intentionally used the term "immutable" and gave you the explanation of it, what I meant. Here described how it works: https://docs.gitlab.com/ee/ci/yaml/README.html#services.
acd
CC BY-SA 4.0
2020-04-01T19:28:22.443Z
you also asked for logs. Here is:  failed to build LLB: rpc error: code = Canceled desc = grpc: the client connection is closing
acd
CC BY-SA 4.0
2020-04-01T19:49:48.033Z
@acd Is that error the same for all three options? What did your yml file look like when you implemented each option? It's best to update your question with what you tried.
BMitch
CC BY-SA 4.0
2020-03-27T00:07:46.760Z
2020-03-27T00:14:09.653Z
2020-03-27T00:14:09.653Z
BMitch
<p>Though it's not the direct answer to my question, but it's a goal, I was aimed at.</p>

<p>I tested several image build tools and stopped on 'Makisu', for building Docker images. It really works fine for me.</p>

<p>The details are here:
<a href="https://github.com/uber/makisu" rel="nofollow noreferrer">https://github.com/uber/makisu</a></p>

<p>and here (starting from 'Using Makisu'):
<a href="https://findwork.dev/blog/6-ways-speed-your-ci/" rel="nofollow noreferrer">https://findwork.dev/blog/6-ways-speed-your-ci/</a></p>

CC BY-SA 4.0
2023-07-25T22:02:37.503Z
Makisu is deprecated as of May 2021. [Kaniko](https://github.com/GoogleContainerTools/kaniko) is a similar tool still being developed. [Here are the GitLab Docs for it](https://docs.gitlab.com/ee/ci/docker/using_kaniko.html)
Nick
CC BY-SA 4.0
2020-04-21T18:03:52.033Z
2020-04-21T18:03:52.033Z
acd
<p>I use GitLab's pipelines to describe the deployment process. And, as part of it, I build Docker images. For that purpose, I use Docker-in-Docker to build those images (i.e. download the <code>Docker-in-Docker</code> image and do all the Docker-stuff, inside of it).</p>

<p>I also set to use Docker Buildkit features. And one of them is <code>--cache-from</code> option (which decreases build time). Which seems doesn't work with <code>Docker-in-Docker</code> (at least, for me).</p>

<p>Here's the pipeline's config:</p>

<pre><code>stages:
  - build
  - purge
  - deploy

variables:
  DOCKER_HOST: tcp://localhost:2375
  DOCKER_TLS_CERTDIR: ""
  DOCKER_BUILDKIT: 1
  LATEST: ${CI_REGISTRY_IMAGE}/${CI_PROJECT_NAME}:${CI_COMMIT_REF_SLUG}
  IMAGE_COMMIT_TAG: ${CI_REGISTRY_IMAGE}/${CI_PROJECT_NAME}:${CI_COMMIT_SHORT_SHA}

services:
  - docker:dind

build:
    stage: build
    image: docker:stable
    script:
        - docker login -u gitlab-ci-token -p ${CI_BUILD_TOKEN} ${CI_REGISTRY}
        - docker build --cache-from ${LATEST} -t ${LATEST} -t ${IMAGE_COMMIT_TAG} .
        - docker push ${IMAGE_COMMIT_TAG}
        - docker push ${LATEST}
</code></pre>

<p>The log output shows the <code>BuildKit</code> is engaged, but there's nothing about using of 'CACHE'. And, the result of it -- 'build' stage takes much more time.</p>

<p>Please, your suggestions, ideas?</p>

<docker><gitlab><gitlab-ci-runner><docker-in-docker>
Build Docker images with DinD and BuildKit
acd
# Build Docker images with DinD and BuildKit

(Asked by: username_0 on 2020-03-26)

I use GitLab's pipelines to describe the deployment process. And, as part of it, I build Docker images. For that purpose, I use Docker-in-Docker to build those images (i.e. download the Docker-in-Docker image and do all the Docker-stuff, inside of it).
I also set to use Docker Buildkit features. And one of them is --cache-from option (which decreases build time). Which seems doesn't work with Docker-in-Docker (at least, for me).
Here's the pipeline's config:
stages:
  - build
  - purge
  - deploy

variables:
  DOCKER_HOST: tcp://localhost:2375
  DOCKER_TLS_CERTDIR: ""
  DOCKER_BUILDKIT: 1
  LATEST: ${CI_REGISTRY_IMAGE}/${CI_PROJECT_NAME}:${CI_COMMIT_REF_SLUG}
  IMAGE_COMMIT_TAG: ${CI_REGISTRY_IMAGE}/${CI_PROJECT_NAME}:${CI_COMMIT_SHORT_SHA}

services:
  - docker:dind

build:
    stage: build
    image: docker:stable
    script:
        - docker login -u gitlab-ci-token -p ${CI_BUILD_TOKEN} ${CI_REGISTRY}
        - docker build --cache-from ${LATEST} -t ${LATEST} -t ${IMAGE_COMMIT_TAG} .
        - docker push ${IMAGE_COMMIT_TAG}
        - docker push ${LATEST}

The log output shows the BuildKit is engaged, but there's nothing about using of 'CACHE'. And, the result of it -- 'build' stage takes much more time.
Please, your suggestions, ideas?

**Tags:** docker, gitlab, gitlab-ci-runner, docker-in-docker


---

## 2 answers

### Answer by username_1 on 2020-03-27. Score: 4

Caching of layers with buildkit in an external registry requires an extra step or two depending on how you want to cache your layers. The easy option is to include a build arg that enables the inline cache:
docker build --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from ${LATEST} -t ${LATEST} -t ${IMAGE_COMMIT_TAG} .

Note that the inline cache only caches layers for the target stage that was pushed in the image, so other states in a multi-stage build would need to be built and cached separately, or rebuilt without caching, neither of which is ideal.
You can also cache to a local file, or push the cache to a different registry image rather than inline with the image you pushed. Unfortunately the standard docker build CLI doesn't have access to all the buildkit flags to enable this. Instead, you can install buildkit directly or use buildx which is a CLI plugin for managing buildkit. You can install buildx separately but in current releases it's available with experimental CLI options that can be enabled with export DOCKER_CLI_EXPERIMENTAL=enabled. I believe you'd need to create a container based builder for all of the buildkit options, which can be done with:
docker buildx create --use --driver docker-container --name local ${DOCKER_HOST:-unix:///var/run/docker.sock}

Then you should be able to run something like:
docker buildx build --cache-from ${IMG_CACHE} --cache-to ${IMG_CACHE} -t ${LATEST} -t ${IMAGE_COMMIT_TAG} .

More documentation on the buildkit caching is available at: https://github.com/moby/buildkit#export-cache
More details on buildx is available at: https://github.com/docker/buildx

> **Comments:**
> **username_0**: HI, @username_1! Sorry for such a late response (I was offline up to now) and thank you for your answer.  I know about `buildx`, but it's not applicable, as I don't have any Docker stuff persistent. I have immutable GL-runner :( and that's why I have to use Docker as service. They same reason is about using external cache.  
> **username_0**: BTW, when I run the build-stage without `DinD` (on different GL-runner) and any external cache, Docker uses cache from the built images not its own build-cache.  
> **username_1**: @username_0 an immutable runner would not be able to build an image. Perhaps you mean an ephemeral runner, which should not prevent you from running docker commands.  
> **username_0**: No-no. "Immutable runner" means persistent instance, which can't be changed, that is to setup `buildkitd`, `buildx`, as a Docker extension, etc.. And, Docker is ephemeral, which is Docker-in-Docker, used in `services` of .gitlab-ci.  
> **username_1**: @username_0 I'm not finding much in my search for gitlab immutable runner, perhaps you're referring to a shared runner? Immutable would mean no write access to /var/lib/docker which would mean all image builds would fail because of a read-only filesystem. It's also not clear from the comments that you attempted all three of the options suggested above. Output of the errors would help.  
> **username_0**: Sorry for confusion. I intentionally used the term "immutable" and gave you the explanation of it, what I meant. Here described how it works: https://docs.gitlab.com/ee/ci/yaml/README.html#services.  
> **username_0**: you also asked for logs. Here is:  failed to build LLB: rpc error: code = Canceled desc = grpc: the client connection is closing  
> **username_1**: @username_0 Is that error the same for all three options? What did your yml file look like when you implemented each option? It's best to update your question with what you tried.  

---

### Answer by username_0 on 2020-04-21. Score: 0

Though it's not the direct answer to my question, but it's a goal, I was aimed at.

I tested several image build tools and stopped on 'Makisu', for building Docker images. It really works fine for me.

The details are here:
<https://github.com/uber/makisu>

and here (starting from 'Using Makisu'):
<https://findwork.dev/blog/6-ways-speed-your-ci/>

> **Comments:**
> **username_2**: Makisu is deprecated as of May 2021. [Kaniko](https://github.com/GoogleContainerTools/kaniko) is a similar tool still being developed. [Here are the GitLab Docs for it](https://docs.gitlab.com/ee/ci/docker/using_kaniko.html)  

---

