<p>Adjust Jenkins job &quot;Build Environment&quot;
<a href="https://i.sstatic.net/m83pi.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/m83pi.png" alt="enter image description here" /></a>
see <a href="https://stackoverflow.com/questions/39829229/how-to-use-exclude-folder-option-in-delete-workspace-before-build-starts-secti/41728663">https://stackoverflow.com/questions/39829229/how-to-use-exclude-folder-option-in-delete-workspace-before-build-starts-secti/41728663</a></p>
<p>also add build step with shell/windows script with</p>
<pre><code>npm install -D
</code></pre>

CC BY-SA 4.0
2020-12-22T16:57:49.017Z
What does the `-D` do on that command? Also how did you get the "Build Environment" configuration section? I dont' see it.
justin.m.chase
CC BY-SA 4.0
2020-09-18T16:55:37.713Z
2020-09-18T20:26:26.087Z
2020-09-18T20:26:26.087Z
Sasha Bond
<p>The best way I have found so far is to use Docker containers with a docker file that does the installation of node_modules before running other steps.</p>
<p>I make sure all jobs of this multi branch pipeline run on the same node and keep the latest docker image for EACH branch on this node locally. That way unless the dependencies in packages.json change, then docker will use the cache to build the new image (or just generate the artifact you want to create and transfer it to the local workspace). If all branches have the same dependencies, they will all use the same cached image, if not they will just create a new one.</p>
<p>Example:</p>
<pre><code>## CHOSE YOUR LOCAL BASE IMAGE
FROM debian:10.0

### ADD YOUR DEPENDENCIES
ADD packages.json build-deps.sh

### INSTALL DEPENDENCIES (we use a custom script)
RUN build-deps.sh

### ADD THE REST OF YOUR CODE 
### This will break the cache assuming your code changes each time,
### but previous docker build commands will be cached 
### (if your dependencies didn't change)
ADD my-code.tar.gz .

### RUN YOUR COMMANDS
RUN run-your-scripts.sh

</code></pre>
<p>After building you image with docker build, you can then do soemthing like this to extract the artifact</p>
<pre><code>docker run --name SOME-NAME YOUR-REPO:YOUR-TAG
sudo docker cp SOME-NAME:/path/to/artifact/artifact.tar.gz ${WORKSPACE}
</code></pre>
<p>I hope this helps</p>

CC BY-SA 4.0
2021-04-29T21:06:28.740Z
2021-04-29T21:22:53.713Z
2021-04-29T21:22:53.713Z
Pedro Graber
<p>It is possible to do this (not with any built-in steps; you essentially have to either use a global cache or write your own caching tool), but I do not recommend it.  When I've tried this in the past, I often ran into race conditions.  For example, one build would write to the cache while another build simultaneously reads from the cache, or two builds would write to the cache simultaneously, etc.  I got a lot of builds with corrupted modules as a result.</p>

<p>Caches do not have these kinds of race conditions when limited to a single job (or in this case, a single branch for a multibranch job) because Jenkins can be configured not to run multiple builds of one job simultaneously.  If you take extra steps to ensure that all of the jobs which might access the cache can't run simultaneously, then you could probably get away with a shared cache.  But at that point, you're likely to lose the added speed benefit of sharing the cache anyway, since you've massively reduced Jenkins' capacity to run jobs in parallel.</p>

<p>The other scenario where you could get away with a shared cache is if you write to the cache extremely infrequently - and I really do mean <em>extremely</em>.  If the cache is rarely written to, then there simply aren't many opportunities for these kinds of race conditions.</p>

CC BY-SA 4.0
2019-06-08T22:39:46.953Z
Great explanation! I never could imagine the problem about race conditions but makes a lot of sense. I'll think another way to handle the installation time instead of creating a cache folder.
Pedro Arantes
CC BY-SA 4.0
2019-06-08T18:03:07.123Z
2019-06-08T18:03:07.123Z
jayhendren
<p>I'm using Jenkins Multibranch Pipeline to handle my CI/CD and I'd like to know if is there a way to cache my <code>node_modules</code> folder?</p>

<p>Every time I push a branch, I execute <code>yarn</code> to install the dependencies. when is <code>master</code> branch, it took 1s to execute this step because it's already cached. Otherwise, when is a new branch, <code>yarn</code> took 2 minutes because is a new environment. Is there a way to cache <code>node_modules</code> globally to be used by all branches/environments?</p>

<jenkins><node.js><cache>
How to cache node_modules folder when using Jenkins Multibranch Pipeline
Pedro Arantes
# How to cache node_modules folder when using Jenkins Multibranch Pipeline

(Asked by: username_0 on 2019-06-08)

I'm using Jenkins Multibranch Pipeline to handle my CI/CD and I'd like to know if is there a way to cache my `node_modules` folder?

Every time I push a branch, I execute `yarn` to install the dependencies. when is `master` branch, it took 1s to execute this step because it's already cached. Otherwise, when is a new branch, `yarn` took 2 minutes because is a new environment. Is there a way to cache `node_modules` globally to be used by all branches/environments?

**Tags:** jenkins, node.js, cache


---

## 3 answers

### Answer by username_4 on 2019-06-08. Score: 5 | Accepted answer

It is possible to do this (not with any built-in steps; you essentially have to either use a global cache or write your own caching tool), but I do not recommend it. When I've tried this in the past, I often ran into race conditions. For example, one build would write to the cache while another build simultaneously reads from the cache, or two builds would write to the cache simultaneously, etc. I got a lot of builds with corrupted modules as a result.

Caches do not have these kinds of race conditions when limited to a single job (or in this case, a single branch for a multibranch job) because Jenkins can be configured not to run multiple builds of one job simultaneously. If you take extra steps to ensure that all of the jobs which might access the cache can't run simultaneously, then you could probably get away with a shared cache. But at that point, you're likely to lose the added speed benefit of sharing the cache anyway, since you've massively reduced Jenkins' capacity to run jobs in parallel.

The other scenario where you could get away with a shared cache is if you write to the cache extremely infrequently - and I really do mean *extremely*. If the cache is rarely written to, then there simply aren't many opportunities for these kinds of race conditions.

> **Comments:**
> **username_0**: Great explanation! I never could imagine the problem about race conditions but makes a lot of sense. I'll think another way to handle the installation time instead of creating a cache folder.  

---

### Answer by username_1 on 2020-09-18. Score: 2

Adjust Jenkins job "Build Environment"

see https://stackoverflow.com/questions/39829229/how-to-use-exclude-folder-option-in-delete-workspace-before-build-starts-secti/41728663
also add build step with shell/windows script with
npm install -D

> **Comments:**
> **username_2**: What does the `-D` do on that command? Also how did you get the "Build Environment" configuration section? I dont' see it.  

---

### Answer by username_3 on 2021-04-29. Score: 2

The best way I have found so far is to use Docker containers with a docker file that does the installation of node_modules before running other steps.
I make sure all jobs of this multi branch pipeline run on the same node and keep the latest docker image for EACH branch on this node locally. That way unless the dependencies in packages.json change, then docker will use the cache to build the new image (or just generate the artifact you want to create and transfer it to the local workspace). If all branches have the same dependencies, they will all use the same cached image, if not they will just create a new one.
Example:
## CHOSE YOUR LOCAL BASE IMAGE
FROM debian:10.0

### ADD YOUR DEPENDENCIES
ADD packages.json build-deps.sh

### INSTALL DEPENDENCIES (we use a custom script)
RUN build-deps.sh

### ADD THE REST OF YOUR CODE 
### This will break the cache assuming your code changes each time,
### but previous docker build commands will be cached 
### (if your dependencies didn't change)
ADD my-code.tar.gz .

### RUN YOUR COMMANDS
RUN run-your-scripts.sh


After building you image with docker build, you can then do soemthing like this to extract the artifact
docker run --name SOME-NAME YOUR-REPO:YOUR-TAG
sudo docker cp SOME-NAME:/path/to/artifact/artifact.tar.gz ${WORKSPACE}

I hope this helps


---

