<p>My solution to this problem was two fold.</p>
<p><strong>First</strong>: As mentioned in the question, I was able to reduce the amount of concurrent Gitlab Instance runners to one, to avoid the usage of too many resource which was causing the shared runner freeze. To do this i moved all my integration tests into a single job. This didn't increase the time of the tests significantly because the largest time cost was the Gitlab instance setup.</p>
<p><strong>Second</strong>: I utilized the retry functionality of the docker image, to respin the CI job if there was an error. I set the retry property to 2, that way it would catch erroneous instance issues, but still fail if a consistent error occurred.</p>
<p>This approach seems to be working well so far! Hopefully it holds out.</p>

CC BY-SA 4.0
2021-02-07T20:20:40.553Z
2021-02-07T20:20:40.553Z
Justin Dalrymple
<p>Can you explain the reason for why the &quot;step_script&quot; installs the dependencies and not use a prebuild images that has all the dependencies installed?</p>
<h3>Solution</h3>
<p>Based on the information you provided, Your test infrastucture rebuild even the test container from scratch.</p>
<p>You tests take a long time to be done as those lines indicate also a fresh composing of the docker container:</p>
<blockquote>
<p>(1/11) Installing ca-certificates (20191127-r2) <br />
(2/11) Installing nghttp2-libs (1.40.0-r1) <br />
(3/11) Installing libcurl (7.67.0-r3) <br />
(4/11) Installing expat (2.2.9-r1) <br />
(5/11) Installing pcre2 (10.34-r1) <br />
(6/11) Installing git (2.24.3-r0) <br />
(7/11) Installing c-ares (1.15.0-r0) <br />
(8/11) Installing libgcc (9.3.0-r0) <br />
(9/11) Installing libstdc++ (9.3.0-r0) <br />
(10/11) Installing nodejs (12.20.1-r0) <br />
(11/11) Installing yarn (1.19.2-r0)</p>
</blockquote>
<p>If you build a docker image with those thing preinstalled, your Tests will run much faster.</p>
<p>This Process creates clean environment where the only change to it will be the code that has changed!!</p>
<p>The new Docker image then need to be only pulled ones and can be reused.</p>
<h1>Build/test infra</h1>
<p><code>docker run -v &quot;$(pwd)&quot;/target:/app:ro -name BUILDID jdalrymple/docker-compose-with-node yarn test</code></p>
<p>run that for each of your tests.</p>

CC BY-SA 4.0
2021-01-21T18:36:46.847Z
Oh nice catch! Ill try that. I'm not sure if it will solve the inconsistent start up of the GitLab container, but who knows. Ill test it out and follow up.
Justin Dalrymple
CC BY-SA 4.0
2021-01-21T18:43:32.817Z
check out https://gitlab.com/jdalrymple/gitlab-bug/-/jobs/970772969 and collaps all sections. on the right side it says how long it took to execute this step. this case 10:15, which translates to 10 mins and 14 secs
Max N
CC BY-SA 4.0
2021-01-21T18:48:31.067Z
It will fix your slow test problem as about 90% is spend on unnessary setup!!!!!
Max N
CC BY-SA 4.0
2021-01-21T18:56:41.473Z
But wouldnt that not solve the issue of the resources required by the running gitlab instance? 

OH! true, how should i modify the DinD image?? Right now, im modifying the image for the stage: docker/compose:latest, with an image that already includes yarn node etc. What should I also do to the DinD image?
Justin Dalrymple
CC BY-SA 4.0
2021-01-21T19:00:12.633Z
Ignore the DinD image for now!!!!!
Max N
CC BY-SA 4.0
2021-01-21T19:24:08.220Z
do you have docker on your dev machine???
Max N
CC BY-SA 4.0
2021-01-21T19:31:48.683Z
did you pass the login data for github to docker correctly????
Max N
CC BY-SA 4.0
2021-01-22T05:15:34.617Z
Yup! Just fixed that. This decreased the setup time by 1-3 minutes! Great idea!!, but it doesn't change the fact that the running GitLab docker instance EATS resources, and running multiples of them causes the crash. https://gitlab.com/jdalrymple/gitlab-bug/-/jobs/979313636
Justin Dalrymple
CC BY-SA 4.0
2021-01-22T11:18:08.323Z
Why are you running docker in docker? With a base Image the code and the build Artifacts are the only changed files.
Max N
CC BY-SA 4.0
2021-01-22T11:33:58.477Z
Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/118781/discussion-between-max-n-and-justin-dalrymple).
Max N
CC BY-SA 4.0
2021-01-21T18:33:19.057Z
2021-01-22T11:41:11.820Z
2021-01-22T11:41:11.820Z
Max N
<p><a href="https://gitlab.com/jdalrymple/gitbeaker" rel="nofollow noreferrer">This</a> project has a pipeline that runs integration tests against a spun up instance of Gitlab. Each Job in the integration test stage has its own instance testing different build distributions (es, cjs, original source). Unfortunately, the shared runners have limited resources and inconsistently cause these jobs to fail because the spun up GitLab instances have insufficient resources. This occurs more often when I have more than one instance spin up at a time since the jobs run in parallel.</p>
<p>Here is a <a href="https://gitlab.com/jdalrymple/gitlab-bug/" rel="nofollow noreferrer">test repo</a> showcasing the problem and it's <a href="https://gitlab.com/jdalrymple/gitlab-bug/-/pipelines/243463960" rel="nofollow noreferrer">pipeline</a></p>
<p>Gitlab Support has suggested using a dedicated runner OR refactoring my pipeline. Since I'm trying to maintain a $0 cost for this open-source project, the only feasible suggestion (that I'm aware of) is the latter one.</p>
<p>I've tested doing just this, specifically splitting these integration tests into separate stages to stop creating multiple GitLab instances at the same time. This, however, has not improved the results. The only other idea I can think of is running all the integration tests in the same script, instead of separate stages or jobs, which would only require spinning up one instance of GitLab. In this solution, I would have to add some sort of id's to the data being created in the tests to avoid having clashes (i.e., creating a project must use a unique name)</p>
<p>What I want to figure out is if I'm overlooking something? Are there any better ways to improve the pipeline to avoid the resource bottlenecks causing the crashing?</p>

CC BY-SA 4.0
2021-01-21T15:48:35.443Z
Wouldn't `test:integration:browser:` also need to include the `&integration` template?
Squirrelkiller
CC BY-SA 4.0
2021-01-21T15:53:01.160Z
Also the test project gitlab-bug you created looks like the pipelines work great, so what's the problem here?
Squirrelkiller
CC BY-SA 4.0
2021-01-21T17:41:34.990Z
@Squirrelkiller The browser test currently doesn't test against a running instance of Gitlab. As for the gitlab-bug repo, the results are inconsistent. If you look at the pipelines list, they fail randomly due to the Gitlab instances not having enough resources when they are spun up. https://gitlab.com/jdalrymple/gitlab-bug/-/pipelines

So im looking for ideas to better manage the pipeline resources
Justin Dalrymple
<gitlab><gitlab-ci-runner>
What would be the best way to improve this Gitlab CI/CD (Gitlab.com) pipeline's efficiency and consistency?
Justin Dalrymple
# What would be the best way to improve this Gitlab CI/CD (Gitlab.com) pipeline's efficiency and consistency?

(Asked by: username_0 on 2021-01-20)

[This](https://gitlab.com/jdalrymple/gitbeaker) project has a pipeline that runs integration tests against a spun up instance of Gitlab. Each Job in the integration test stage has its own instance testing different build distributions (es, cjs, original source). Unfortunately, the shared runners have limited resources and inconsistently cause these jobs to fail because the spun up GitLab instances have insufficient resources. This occurs more often when I have more than one instance spin up at a time since the jobs run in parallel.

Here is a [test repo](https://gitlab.com/jdalrymple/gitlab-bug/) showcasing the problem and it's [pipeline](https://gitlab.com/jdalrymple/gitlab-bug/-/pipelines/243463960)

Gitlab Support has suggested using a dedicated runner OR refactoring my pipeline. Since I'm trying to maintain a $0 cost for this open-source project, the only feasible suggestion (that I'm aware of) is the latter one.

I've tested doing just this, specifically splitting these integration tests into separate stages to stop creating multiple GitLab instances at the same time. This, however, has not improved the results. The only other idea I can think of is running all the integration tests in the same script, instead of separate stages or jobs, which would only require spinning up one instance of GitLab. In this solution, I would have to add some sort of id's to the data being created in the tests to avoid having clashes (i.e., creating a project must use a unique name)

What I want to figure out is if I'm overlooking something? Are there any better ways to improve the pipeline to avoid the resource bottlenecks causing the crashing?

**Tags:** gitlab, gitlab-ci-runner

> **Comments:**
> **username_1**: Wouldn't `test:integration:browser:` also need to include the `&integration` template?  
> **username_1**: Also the test project gitlab-bug you created looks like the pipelines work great, so what's the problem here?  
> **username_0**: @username_1 The browser test currently doesn't test against a running instance of Gitlab. As for the gitlab-bug repo, the results are inconsistent. If you look at the pipelines list, they fail randomly due to the Gitlab instances not having enough resources when they are spun up. https://gitlab.com/jdalrymple/gitlab-bug/-/pipelines  So im looking for ideas to better manage the pipeline resources  

---

## 2 answers

### Answer by username_0 on 2021-02-07. Score: 0 | Accepted answer

My solution to this problem was two fold.

**First**: As mentioned in the question, I was able to reduce the amount of concurrent Gitlab Instance runners to one, to avoid the usage of too many resource which was causing the shared runner freeze. To do this i moved all my integration tests into a single job. This didn't increase the time of the tests significantly because the largest time cost was the Gitlab instance setup.

**Second**: I utilized the retry functionality of the docker image, to respin the CI job if there was an error. I set the retry property to 2, that way it would catch erroneous instance issues, but still fail if a consistent error occurred.

This approach seems to be working well so far! Hopefully it holds out.


---

### Answer by username_2 on 2021-01-21. Score: 1

Can you explain the reason for why the "step\_script" installs the dependencies and not use a prebuild images that has all the dependencies installed?

### Solution

Based on the information you provided, Your test infrastucture rebuild even the test container from scratch.

You tests take a long time to be done as those lines indicate also a fresh composing of the docker container:

> (1/11) Installing ca-certificates (20191127-r2)   
> (2/11) Installing nghttp2-libs (1.40.0-r1)   
> (3/11) Installing libcurl (7.67.0-r3)   
> (4/11) Installing expat (2.2.9-r1)   
> (5/11) Installing pcre2 (10.34-r1)   
> (6/11) Installing git (2.24.3-r0)   
> (7/11) Installing c-ares (1.15.0-r0)   
> (8/11) Installing libgcc (9.3.0-r0)   
> (9/11) Installing libstdc++ (9.3.0-r0)   
> (10/11) Installing nodejs (12.20.1-r0)   
> (11/11) Installing yarn (1.19.2-r0)

If you build a docker image with those thing preinstalled, your Tests will run much faster.

This Process creates clean environment where the only change to it will be the code that has changed!!

The new Docker image then need to be only pulled ones and can be reused.

# Build/test infra

`docker run -v "$(pwd)"/target:/app:ro -name BUILDID jdalrymple/docker-compose-with-node yarn test`

run that for each of your tests.

> **Comments:**
> **username_0**: Oh nice catch! Ill try that. I'm not sure if it will solve the inconsistent start up of the GitLab container, but who knows. Ill test it out and follow up.  
> **username_2**: check out https://gitlab.com/jdalrymple/gitlab-bug/-/jobs/970772969 and collaps all sections. on the right side it says how long it took to execute this step. this case 10:15, which translates to 10 mins and 14 secs  
> **username_2**: It will fix your slow test problem as about 90% is spend on unnessary setup!!!!!  
> **username_0**: But wouldnt that not solve the issue of the resources required by the running gitlab instance?   OH! true, how should i modify the DinD image?? Right now, im modifying the image for the stage: docker/compose:latest, with an image that already includes yarn node etc. What should I also do to the DinD image?  
> **username_2**: Ignore the DinD image for now!!!!!  
> **username_2**: do you have docker on your dev machine???  
> **username_2**: did you pass the login data for github to docker correctly????  
> **username_0**: Yup! Just fixed that. This decreased the setup time by 1-3 minutes! Great idea!!, but it doesn't change the fact that the running GitLab docker instance EATS resources, and running multiples of them causes the crash. https://gitlab.com/jdalrymple/gitlab-bug/-/jobs/979313636  
> **username_2**: Why are you running docker in docker? With a base Image the code and the build Artifacts are the only changed files.  
> **username_2**: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/118781/discussion-between-max-n-and-justin-dalrymple).  

---

