<p>Perhaps you are looking for the <a href="https://plugins.jenkins.io/jobcacher" rel="nofollow noreferrer">Job Cacher</a> plugin?</p>
<p>To quote the plugin page:</p>
<blockquote>
<p><strong>Summary</strong></p>
<p>Enables caching of files on transient executors for use on subsequent builds to improve performance</p>
<p><strong>Features</strong></p>
<ul>
<li>Item storage extension point supporting on master storage and AWS S3</li>
<li>Cache Wrapper for free style jobs that manages the cache
Arbitrary File Cache where user specifies paths to be cached</li>
<li>UI on Job page to review the job's caches</li>
<li>Supports Pipeline jobs with a cache directive</li>
<li>Cache Extension Point for other plugins to provide opinionated caching capability such as Gradle caches</li>
<li>Incremental storing of cache into storage to minimize data movement</li>
</ul>
</blockquote>
<p>However, these are for files <strong>created</strong> by the jobs, not <strong>needed</strong> by them. In order to use it in your case, you might need a few dummy jobs specifically for downloading the content from the arbitrary URLs, and then caching them in your Jenkins cache. I guess you'd have to write the script for that.</p>
<p>To be frank, I've never used this plugin, and what I describe above doesn't seem to be an elegant solution. Your question specifically ask for a Jenkins plugin, however I believe the <em>general</em> problem is <strong>how to cache job dependencies</strong> in Jenkins. If you agree to this wider interpretation, there are some other options.</p>
<p>I have often solved the problem you mention using a combination of managed scripts and persistent storage which Jenkins and workers have access to, in Jenkins.  If you need really <em>arbitrary</em> dependency management, a tool like <a href="https://spack.readthedocs.io" rel="nofollow noreferrer">Spack</a> or <a href="https://easybuild.readthedocs.io/en/latest/" rel="nofollow noreferrer">EasyBuild</a> may be useful. Although these have evolved in the HPC domain, they do quite well at caching dependencies for workloads.</p>
<p>However, since you mention just a few dependencies (specific sqlite versions <em>e.g.</em>), I think it may be worth your time to write a script which does the caching for you and, together with globally-available persistent storage, do the caching and retrieval that way.</p>

CC BY-SA 4.0
2019-11-05T05:27:45.183Z
That seems like a means to cache _my own_ artifact -- file produced by my build. What I'm looking for is a way to efficiently download somebody else's tarball -- from afar... I know, I can write a script of my own, but I'm puzzled, why -- given their wide variety -- there is no plugin for this among the existing ones. FreeBSD's ports-collection's been doing just this (downloading, verifying, and building 3rd-party sources) for decades, for example...
Mikhail T.
CC BY-SA 4.0
2019-11-04T06:46:31.560Z
2019-11-04T06:46:31.560Z
2020-06-18T08:28:10.803Z
Bruce Becker
<p>Although <em>our own</em> sources are checked out from a revision-control systems, the third-party software we depend on needs to be downloaded from their respective sites.</p>

<p>The plugin I'm looking for would offer a "Source Management" (or "Pre-Build") step, where I could register one (or more) sources for the same file. For example:</p>

<ul>
<li><a href="https://www.sqlite.org/2019/sqlite-autoconf-3290000.tar.gz" rel="nofollow noreferrer">https://www.sqlite.org/2019/sqlite-autoconf-3290000.tar.gz</a></li>
<li><a href="http://www2.sqlite.org/2019/sqlite-autoconf-3290000.tar.gz" rel="nofollow noreferrer">http://www2.sqlite.org/2019/sqlite-autoconf-3290000.tar.gz</a></li>
<li><a href="http://www3.sqlite.org/2019/sqlite-autoconf-3290000.tar.gz" rel="nofollow noreferrer">http://www3.sqlite.org/2019/sqlite-autoconf-3290000.tar.gz</a></li>
</ul>

<p>and its digest (such as SHA256).</p>

<p>At build- (or at a prebuild-) time, it would check its cache and, if the file is not present there, download and verify it, and make it available to the subsequent build-steps.</p>

<p>I'm sure something like this, or very similar, exists already, but I cannot find it. Sonatype's Nexus seems to offer this, but only for files already found in one's Nexus repository, not for arbitrary URL's.</p>

<p>Any help?</p>

<jenkins><jenkins-plugins>
Is there a Jenkins plugin for downloading source code?
Mikhail T.
# Is there a Jenkins plugin for downloading source code?

(Asked by: username_0 on 2019-11-03)

Although *our own* sources are checked out from a revision-control systems, the third-party software we depend on needs to be downloaded from their respective sites.

The plugin I'm looking for would offer a "Source Management" (or "Pre-Build") step, where I could register one (or more) sources for the same file. For example:

* <https://www.sqlite.org/2019/sqlite-autoconf-3290000.tar.gz>
* <http://www2.sqlite.org/2019/sqlite-autoconf-3290000.tar.gz>
* <http://www3.sqlite.org/2019/sqlite-autoconf-3290000.tar.gz>

and its digest (such as SHA256).

At build- (or at a prebuild-) time, it would check its cache and, if the file is not present there, download and verify it, and make it available to the subsequent build-steps.

I'm sure something like this, or very similar, exists already, but I cannot find it. Sonatype's Nexus seems to offer this, but only for files already found in one's Nexus repository, not for arbitrary URL's.

Any help?

**Tags:** jenkins, jenkins-plugins


---

## 1 answers

### Answer by username_1 on 2019-11-04. Score: 1

Perhaps you are looking for the [Job Cacher](https://plugins.jenkins.io/jobcacher) plugin?

To quote the plugin page:

> **Summary**
>
> Enables caching of files on transient executors for use on subsequent builds to improve performance
>
> **Features**
>
> * Item storage extension point supporting on master storage and AWS S3
> * Cache Wrapper for free style jobs that manages the cache
>   Arbitrary File Cache where user specifies paths to be cached
> * UI on Job page to review the job's caches
> * Supports Pipeline jobs with a cache directive
> * Cache Extension Point for other plugins to provide opinionated caching capability such as Gradle caches
> * Incremental storing of cache into storage to minimize data movement

However, these are for files **created** by the jobs, not **needed** by them. In order to use it in your case, you might need a few dummy jobs specifically for downloading the content from the arbitrary URLs, and then caching them in your Jenkins cache. I guess you'd have to write the script for that.

To be frank, I've never used this plugin, and what I describe above doesn't seem to be an elegant solution. Your question specifically ask for a Jenkins plugin, however I believe the *general* problem is **how to cache job dependencies** in Jenkins. If you agree to this wider interpretation, there are some other options.

I have often solved the problem you mention using a combination of managed scripts and persistent storage which Jenkins and workers have access to, in Jenkins. If you need really *arbitrary* dependency management, a tool like [Spack](https://spack.readthedocs.io) or [EasyBuild](https://easybuild.readthedocs.io/en/latest/) may be useful. Although these have evolved in the HPC domain, they do quite well at caching dependencies for workloads.

However, since you mention just a few dependencies (specific sqlite versions *e.g.*), I think it may be worth your time to write a script which does the caching for you and, together with globally-available persistent storage, do the caching and retrieval that way.

> **Comments:**
> **username_0**: That seems like a means to cache _my own_ artifact -- file produced by my build. What I'm looking for is a way to efficiently download somebody else's tarball -- from afar... I know, I can write a script of my own, but I'm puzzled, why -- given their wide variety -- there is no plugin for this among the existing ones. FreeBSD's ports-collection's been doing just this (downloading, verifying, and building 3rd-party sources) for decades, for example...  

---

