<p>GitLab has some docs on how to speed up builds, and one of the suggestions does involve caching runner images.</p>
<p><a href="https://docs.gitlab.com/runner/configuration/speed_up_job_execution.html#use-a-proxy-for-containers" rel="nofollow noreferrer">https://docs.gitlab.com/runner/configuration/speed_up_job_execution.html#use-a-proxy-for-containers</a></p>
<h2>Use a proxy for containers</h2>
<p>You can speed up the time it takes to download Docker images by using:</p>
<ul>
<li>The GitLab Dependency Proxy or</li>
<li>A mirror of the DockerHub Registry</li>
<li>Other open source solutions</li>
</ul>
<p>There are other non-container related suggestions too, as well as information on how to use these up at that link.</p>
<p>You can also host your own runners, in which case you can run them on docker or just centos, etc.  I assume, in that case, that you could already have the images available on the runner node to avoid new pulls.  We run ours in kubernetes and are looking at harbor as an option to speed these things up too, but we haven't tested it yet.  I imagine self-hosted k8s and harbor would be similar to GitLab using its own dependency proxy solution.</p>

CC BY-SA 4.0
2022-02-26T22:29:20.380Z
My Runner(s) are actually self hosted. They run on a Server as Docker-Runner. If I understand correctly instead of linking a proxy somewhere I could save the images (In my example python:3.9) locally inside the gitlab runner node and reference them?
MaKaNu
CC BY-SA 4.0
2022-02-27T02:28:41.723Z
If you're using a single-server docker setup, I wouldn't expect it to re-pull images it already had.  The underlying docker daemon should recognize it already has an image/relevant layers and should just reuse it without going to docker-hub/etc.  That's how docker works.
John Humphreys
CC BY-SA 4.0
2022-02-27T15:00:35.747Z
So it seems I am doing something wrong in the pipeline, because I just use one Dockerized-Runner on a single server, but both jobs are pulling the python:3.9 image.
MaKaNu
CC BY-SA 4.0
2023-06-05T10:50:52.970Z
The docs you link to involve caching in a registry (even if local) as opposed to the local docker cache, which is what I'm looking for, and what I assume the question is asking for as well.
Chris Stryczynski
CC BY-SA 4.0
2022-02-26T19:09:27.697Z
2022-02-26T19:09:27.697Z
John Humphreys
<p>I try to improve the performance of my pipeline stages in gitlab cicd. The Gitlab is self hosted and the Runner runs as docker with docker executor. I recognized, that for both jobs in my stages (test, build) they begin with pulling the images from docker hub:</p>
<pre class="lang-bash prettyprint-override"><code>Running with gitlab-runner 14.8.0 (565b6c0b)
  on Shared gitlab-runner on ARWS. 3yyScM1M
Preparing the &quot;docker&quot; executor 00:03
Using Docker executor with image python:3.9 ...
Pulling docker image python:3.9 ...
Using docker image sha256:4819be0df94257e1e31cd64dda12d46ff8b2180f9576ad9eaf98dcac9d70d9f9 for python:3.9 with digest python@sha256:57274f3fe86ad3d010025e9019bc41dcf9ee5833cfe4ba7b6773ef49672d855f ...
Preparing environment
.
.
.
</code></pre>
<p>This part seems to take most of the pipeline time. At the moment not much is happening in both jobs.
I already started to implement caching in the pipelinbe:</p>
<pre class="lang-yaml prettyprint-override"><code>image: python:3.9

stages:
  - test
  - build

variables:
  PIP_CACHE_DIR: &quot;$CI_PROJECT_DIR/.cache/pip&quot;

cache:
  key:
    files:
      - poetry.lock
      - .gitlab-ci.yml
    prefix: ${CI_JOB_NAME}
  paths:
    - .venv
    - .cache/pip

before_script:
  - python --version  # For debugging
  - pip install poetry
  - poetry config virtualenvs.in-project true
  - poetry install
  - source `poetry env info --path`/bin/activate

testing:
  tags: 
    - docker
    - linux
  stage: test
  script:
    - echo &quot;Launching Job 'testing' in stage 'test'&quot;
    - pytest -v --junitxml=report.xml
  artifacts:
    when: always
    reports:
      junit: /builds/$CI_PROJECT_PATH/report.xml

building:
  tags:
    - docker
    - linux
  stage: build
  script:
    - echo &quot;Launching Job 'building' in stage 'build'&quot;
</code></pre>
<p>And there is a huge possibility I do some stupid stuff. But the real question bothers me: Is it possible to cache the environment instead of pulling and preparing it at the start of every job?</p>

<docker><gitlab><gitlab-ci-runner><cache>
How to caching the docker images?
MaKaNu
# How to caching the docker images?

(Asked by: username_0 on 2022-02-26)

I try to improve the performance of my pipeline stages in gitlab cicd. The Gitlab is self hosted and the Runner runs as docker with docker executor. I recognized, that for both jobs in my stages (test, build) they begin with pulling the images from docker hub:
Running with gitlab-runner 14.8.0 (565b6c0b)
  on Shared gitlab-runner on ARWS. 3yyScM1M
Preparing the "docker" executor 00:03
Using Docker executor with image python:3.9 ...
Pulling docker image python:3.9 ...
Using docker image sha256:4819be0df94257e1e31cd64dda12d46ff8b2180f9576ad9eaf98dcac9d70d9f9 for python:3.9 with digest python@sha256:57274f3fe86ad3d010025e9019bc41dcf9ee5833cfe4ba7b6773ef49672d855f ...
Preparing environment
.
.
.

This part seems to take most of the pipeline time. At the moment not much is happening in both jobs.
I already started to implement caching in the pipelinbe:
image: python:3.9

stages:
  - test
  - build

variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"

cache:
  key:
    files:
      - poetry.lock
      - .gitlab-ci.yml
    prefix: ${CI_JOB_NAME}
  paths:
    - .venv
    - .cache/pip

before_script:
  - python --version  # For debugging
  - pip install poetry
  - poetry config virtualenvs.in-project true
  - poetry install
  - source `poetry env info --path`/bin/activate

testing:
  tags: 
    - docker
    - linux
  stage: test
  script:
    - echo "Launching Job 'testing' in stage 'test'"
    - pytest -v --junitxml=report.xml
  artifacts:
    when: always
    reports:
      junit: /builds/$CI_PROJECT_PATH/report.xml

building:
  tags:
    - docker
    - linux
  stage: build
  script:
    - echo "Launching Job 'building' in stage 'build'"

And there is a huge possibility I do some stupid stuff. But the real question bothers me: Is it possible to cache the environment instead of pulling and preparing it at the start of every job?

**Tags:** docker, gitlab, gitlab-ci-runner, cache


---

## 1 answers

### Answer by username_1 on 2022-02-26. Score: 0

GitLab has some docs on how to speed up builds, and one of the suggestions does involve caching runner images.

<https://docs.gitlab.com/runner/configuration/speed_up_job_execution.html#use-a-proxy-for-containers>

## Use a proxy for containers

You can speed up the time it takes to download Docker images by using:

* The GitLab Dependency Proxy or
* A mirror of the DockerHub Registry
* Other open source solutions

There are other non-container related suggestions too, as well as information on how to use these up at that link.

You can also host your own runners, in which case you can run them on docker or just centos, etc. I assume, in that case, that you could already have the images available on the runner node to avoid new pulls. We run ours in kubernetes and are looking at harbor as an option to speed these things up too, but we haven't tested it yet. I imagine self-hosted k8s and harbor would be similar to GitLab using its own dependency proxy solution.

> **Comments:**
> **username_0**: My Runner(s) are actually self hosted. They run on a Server as Docker-Runner. If I understand correctly instead of linking a proxy somewhere I could save the images (In my example python:3.9) locally inside the gitlab runner node and reference them?  
> **username_1**: If you're using a single-server docker setup, I wouldn't expect it to re-pull images it already had.  The underlying docker daemon should recognize it already has an image/relevant layers and should just reuse it without going to docker-hub/etc.  That's how docker works.  
> **username_0**: So it seems I am doing something wrong in the pipeline, because I just use one Dockerized-Runner on a single server, but both jobs are pulling the python:3.9 image.  
> **username_2**: The docs you link to involve caching in a registry (even if local) as opposed to the local docker cache, which is what I'm looking for, and what I assume the question is asking for as well.  

---

