<p>I highly recommend using <a href="https://docs.npmjs.com/cli/ci.html" rel="nofollow noreferrer">npm ci</a> instead of <code>npm install</code>. The slow part of installing npm modules is figuring out everything that needs to be installed. If you commit your package-lock.json and use <code>npm ci</code>, it skips that part entirely and just looks at your lockfile to download what's needed. It's significantly faster.</p>

CC BY-SA 4.0
2020-09-24T15:39:58.007Z
What are your thoughts on npm install with `--prefer-offline` ? I was using npm ci but switched to npm i with the usage of prefer offline. As of yet I don't have an opinion on either - other that npm ci FEELS better due do the use of the keyword `ci`
Jacksonkr
CC BY-SA 4.0
2020-09-24T18:00:07.727Z
I've never used `--prefer-offline`. In fact I didn't know it existed before your comment. I think the main benefit of `npm ci` is the fact that you know exactly what will be installed and there are no changes made to package.json. It also deletes node_modules before installing packages so you know it's a clean slate every time.

I've not yet seen issues with `npm ci` slowness, but you could use `--prefer-offline` with `npm ci`. Here's an example from the gitlab docs:  https://docs.gitlab.com/ee/ci/caching/#caching-nodejs-dependencies
bradym
CC BY-SA 4.0
2020-09-25T12:51:43.917Z
Funny you should mention that gitlab doc - those docs are where I pulled most my current implementation from. It seems like the issue now is that compiling / extracting the cache takes longer than fetching all the npm packages.
Jacksonkr
CC BY-SA 4.0
2020-09-25T18:51:46.917Z
Circling back - ultimately I deemed my `prepare` job to be doing double work. I took `prepare` out which cut my workload in half. I don't know I'd call that optimized but it's certainly much better.
Jacksonkr
CC BY-SA 4.0
2020-09-26T00:23:59.570Z
Ahh, yes. I don't think I noticed you were doing both. And optimization is relative, so yes, it's optimized. :D
bradym
CC BY-SA 4.0
2020-09-24T00:33:51.750Z
2020-09-24T00:33:51.750Z
bradym
<p>I have a gitlab yaml I've been working on for a bit and while everything works it seems like each job takes quite a bit longer than it should. I tend to think the culprit is that npm files are fetched every time rather than sharing the <code>node_modules</code> cache between jobs.</p>
<p>Short of creating my own docker image, how can I optimize this yaml in a way that reduces the bottlenecks ?</p>
<p><strong>gitlab .yaml</strong></p>
<pre><code>image: node:10.19.0 # https://hub.docker.com/_/node/
# image: node:latest

cache:
  # untracked: true
  key: my-project
  # key: ${CI_COMMIT_REF_SLUG} # per branch
  # key:
  #   files:
  #     - package-lock.json # only update cache when this file changes (not working) @jkr
  paths:
    - .npm/
    - node_modules
    - build

stages:
  - prepare # prepares builds, makes build needed for testing
  - test # uses test:build specifically @jkr
  - build
  - deploy

# before_install:

before_script:
  - npm ci --cache .npm --prefer-offline

prepare:
  stage: prepare
  needs: []
  script:
    - npm install

test:
  stage: test
  needs: [prepare]
  except:
    - schedules
  tags:
    - linux
  script:
    - npm run build:dev
    - npm run test:cicd-deps
    - npm run test:cicd # runs puppeteer tests @jkr
  artifacts:
    reports:
      junit: junit.xml
    paths:
      - coverage/

build-staging:
  stage: build
  needs: [prepare]
  only:
    - schedules
  before_script:
    - apt-get update &amp;&amp; apt-get install -y zip
  script:
    - npm run build:staging
    - zip -r build-staging.zip build
  # cache:
  #   paths:
  #     - build
  #   &lt;&lt;: *global_cache
  #   policy: push
  artifacts:
    paths:
      - build-staging.zip

deploy-serv1:
  stage: deploy
  needs: [build-staging]
  tags: [linux]
  only:
    - schedules
  #   # - branches@gitlab-org/gitlab
  before_script:
    - apt-get update &amp;&amp; apt-get install -y lftp
  script:
    # temporarily using 'verify-certificate no'
    # for more on verify-certificate @jkr: https://www.versatilewebsolutions.com/blog/2014/04/lftp-ftps-and-certificate-verification.html
    # variables do not work with 'single quotes' unless they are &quot;'surrounded by doubles'&quot;
    - lftp -e &quot;set ssl:verify-certificate no; open myserver.org; user $LFTP_USERNAME $LFTP_PASSWORD; mirror --reverse --verbose build/ /var/www/domains/dev/projects/my-project/build/; bye&quot;
  # environment:
  #   name: staging
  #   url: http://dev.myserver.org/projects/my-project/build
  when: manual
  allow_failure: true

build-production:
  stage: build
  needs: [prepare]
  only:
    - schedules
  before_script:
    - apt-get update &amp;&amp; apt-get install -y zip
  script:
    - npm run build:staging
    - zip -r build-prod.zip build
  # cache:
  #   paths:
  #     - build
  #   &lt;&lt;: *global_cache
  #   policy: push
  artifacts:
    paths:
      - build-prod.zip

deploy-serv2:
  stage: deploy
  needs: [build-production]
  tags: [linux]
  only:
    - schedules
    # - master
  before_script:
    - apt-get update &amp;&amp; apt-get install -y lftp
  script:
    - sh deploy-prod
  environment:
    name: production
    url: http://www.my-project.org
  when: manual
  allow_failure: true
</code></pre>

<continuous-integration><yaml>
devops - Optomizing gitlab cicd pipeline's caching
Jacksonkr
# devops - Optomizing gitlab cicd pipeline's caching

(Asked by: username_0 on 2020-09-23)

I have a gitlab yaml I've been working on for a bit and while everything works it seems like each job takes quite a bit longer than it should. I tend to think the culprit is that npm files are fetched every time rather than sharing the node_modules cache between jobs.
Short of creating my own docker image, how can I optimize this yaml in a way that reduces the bottlenecks ?
gitlab .yaml
image: node:10.19.0 # https://hub.docker.com/_/node/
# image: node:latest

cache:
  # untracked: true
  key: my-project
  # key: ${CI_COMMIT_REF_SLUG} # per branch
  # key:
  #   files:
  #     - package-lock.json # only update cache when this file changes (not working) @jkr
  paths:
    - .npm/
    - node_modules
    - build

stages:
  - prepare # prepares builds, makes build needed for testing
  - test # uses test:build specifically @jkr
  - build
  - deploy

# before_install:

before_script:
  - npm ci --cache .npm --prefer-offline

prepare:
  stage: prepare
  needs: []
  script:
    - npm install

test:
  stage: test
  needs: [prepare]
  except:
    - schedules
  tags:
    - linux
  script:
    - npm run build:dev
    - npm run test:cicd-deps
    - npm run test:cicd # runs puppeteer tests @jkr
  artifacts:
    reports:
      junit: junit.xml
    paths:
      - coverage/

build-staging:
  stage: build
  needs: [prepare]
  only:
    - schedules
  before_script:
    - apt-get update && apt-get install -y zip
  script:
    - npm run build:staging
    - zip -r build-staging.zip build
  # cache:
  #   paths:
  #     - build
  #   <<: *global_cache
  #   policy: push
  artifacts:
    paths:
      - build-staging.zip

deploy-serv1:
  stage: deploy
  needs: [build-staging]
  tags: [linux]
  only:
    - schedules
  #   # - branches@gitlab-org/gitlab
  before_script:
    - apt-get update && apt-get install -y lftp
  script:
    # temporarily using 'verify-certificate no'
    # for more on verify-certificate @jkr: https://www.versatilewebsolutions.com/blog/2014/04/lftp-ftps-and-certificate-verification.html
    # variables do not work with 'single quotes' unless they are "'surrounded by doubles'"
    - lftp -e "set ssl:verify-certificate no; open myserver.org; user $LFTP_USERNAME $LFTP_PASSWORD; mirror --reverse --verbose build/ /var/www/domains/dev/projects/my-project/build/; bye"
  # environment:
  #   name: staging
  #   url: http://dev.myserver.org/projects/my-project/build
  when: manual
  allow_failure: true

build-production:
  stage: build
  needs: [prepare]
  only:
    - schedules
  before_script:
    - apt-get update && apt-get install -y zip
  script:
    - npm run build:staging
    - zip -r build-prod.zip build
  # cache:
  #   paths:
  #     - build
  #   <<: *global_cache
  #   policy: push
  artifacts:
    paths:
      - build-prod.zip

deploy-serv2:
  stage: deploy
  needs: [build-production]
  tags: [linux]
  only:
    - schedules
    # - master
  before_script:
    - apt-get update && apt-get install -y lftp
  script:
    - sh deploy-prod
  environment:
    name: production
    url: http://www.my-project.org
  when: manual
  allow_failure: true

**Tags:** continuous-integration, yaml


---

## 1 answers

### Answer by username_1 on 2020-09-24. Score: 3 | Accepted answer

I highly recommend using [npm ci](https://docs.npmjs.com/cli/ci.html) instead of `npm install`. The slow part of installing npm modules is figuring out everything that needs to be installed. If you commit your package-lock.json and use `npm ci`, it skips that part entirely and just looks at your lockfile to download what's needed. It's significantly faster.

> **Comments:**
> **username_0**: What are your thoughts on npm install with `--prefer-offline` ? I was using npm ci but switched to npm i with the usage of prefer offline. As of yet I don't have an opinion on either - other that npm ci FEELS better due do the use of the keyword `ci`  
> **username_1**: I've never used `--prefer-offline`. In fact I didn't know it existed before your comment. I think the main benefit of `npm ci` is the fact that you know exactly what will be installed and there are no changes made to package.json. It also deletes node_modules before installing packages so you know it's a clean slate every time.  I've not yet seen issues with `npm ci` slowness, but you could use `--prefer-offline` with `npm ci`. Here's an example from the gitlab docs:  https://docs.gitlab.com/ee/ci/caching/#caching-nodejs-dependencies  
> **username_0**: Funny you should mention that gitlab doc - those docs are where I pulled most my current implementation from. It seems like the issue now is that compiling / extracting the cache takes longer than fetching all the npm packages.  
> **username_0**: Circling back - ultimately I deemed my `prepare` job to be doing double work. I took `prepare` out which cut my workload in half. I don't know I'd call that optimized but it's certainly much better.  
> **username_1**: Ahh, yes. I don't think I noticed you were doing both. And optimization is relative, so yes, it's optimized. :D  

---

