<p>You could try to run the artifactory file upload in parallel if you are using the Jenkinsfile syntax: <a href="https://github.com/jenkinsci/pipeline-examples/blob/master/pipeline-examples/parallel-from-list/parallelFromList.groovy" rel="noreferrer">https://github.com/jenkinsci/pipeline-examples/blob/master/pipeline-examples/parallel-from-list/parallelFromList.groovy</a></p>

<p>Here is a simpler example to run things in parallel if you only have a fixed number of things you want to do in parallel:</p>

<pre><code>parallel (
    "task1" : {
        //runTask1
    },
    "task2" : {
        //runTask2
    },
    "task3" : {
        //runTask3
    },
)
</code></pre>

<p>This can be enclosed inside a <code>node</code>, in which case they would all share the same workspace or each command can define <code>node</code> inside the task code to run into another node in which case it would have a different workspace for each task.</p>

CC BY-SA 3.0
2017-05-05T07:40:16.867Z
Hi Michael! I was talking more about downloading files and not uploading. If I did this with downloading, wouldn't it save the files on different nodes rather than a single workspace?
Asaf Haim
CC BY-SA 3.0
2017-05-05T13:03:28.377Z
Not necessarily, I will give you a better example in a few moments
Michael Pereira
CC BY-SA 3.0
2017-05-10T12:38:35.907Z
Not sure that matches my case since when I download artifacts, I don't know the exact number of artifacts (using ant-like syntax to download the entire repository)
Asaf Haim
CC BY-SA 3.0
2017-05-10T15:25:30.280Z
In that case, take a look at https://jenkins.io/doc/pipeline/examples/#jobs-in-parallel You can reuse the same code and just change the loop counter to the number of elements you have to download, and replace the `build job` line with the one to download from artifactory
Michael Pereira
CC BY-SA 3.0
2017-05-14T09:03:11.787Z
But what if I don't know the number of files I download? I use the double asterisk notation (**) to recursively download all my repository, so neither I know how many there are nor their names.
Asaf Haim
CC BY-SA 3.0
2017-05-04T19:20:01.353Z
2017-05-08T16:22:10.220Z
2017-05-08T16:22:10.220Z
Michael Pereira
<p>In my company we're having a bad time trying to download with the Artifactory plugin in Jenkins because we have a lot of tiny files in our repository and apparently the plugin works in a single thread, so it takes a REALLY long time. We tested the JFrog CLI where we can set the number of threads for the download and that actually works great. Still, we really need to use the original plugin.</p>

<p>Is there any workaround for this? Maybe it has something to do with the Artifactory server configuration?</p>

<p><strong>Note</strong>: We have so many small files because that's how the development is being done in the company I'm afraid. We thought about archiving all the files before we store them but that would probably be non-economical in terms of storage, since every uploaded zip file would probably have a different checksum.</p>

CC BY-SA 3.0
2017-05-10T13:19:20.747Z
Typically one would package the application prior to uploading to artifactory.  I was just wondering how was artifactory used in your development process, what is the platform or language used and how are these files consumed.  Having a broader picture of the process could help us find potential solutions that might not be obvious when focusing on the download issue alone.
Newtopian
<jenkins><jenkins-plugins><jenkins-pipeline><artifacts>
Jenkins Artifactory plugin not downloading in multi-thread
Asaf Haim
# Jenkins Artifactory plugin not downloading in multi-thread

(Asked by: username_0 on 2017-05-04)

In my company we're having a bad time trying to download with the Artifactory plugin in Jenkins because we have a lot of tiny files in our repository and apparently the plugin works in a single thread, so it takes a REALLY long time. We tested the JFrog CLI where we can set the number of threads for the download and that actually works great. Still, we really need to use the original plugin.

Is there any workaround for this? Maybe it has something to do with the Artifactory server configuration?

**Note**: We have so many small files because that's how the development is being done in the company I'm afraid. We thought about archiving all the files before we store them but that would probably be non-economical in terms of storage, since every uploaded zip file would probably have a different checksum.

**Tags:** jenkins, jenkins-plugins, jenkins-pipeline, artifacts

> **Comments:**
> **username_1**: Typically one would package the application prior to uploading to artifactory.  I was just wondering how was artifactory used in your development process, what is the platform or language used and how are these files consumed.  Having a broader picture of the process could help us find potential solutions that might not be obvious when focusing on the download issue alone.  

---

## 1 answers

### Answer by username_2 on 2017-05-04. Score: 6

You could try to run the artifactory file upload in parallel if you are using the Jenkinsfile syntax: https://github.com/jenkinsci/pipeline-examples/blob/master/pipeline-examples/parallel-from-list/parallelFromList.groovy
Here is a simpler example to run things in parallel if you only have a fixed number of things you want to do in parallel:
parallel (
    "task1" : {
        //runTask1
    },
    "task2" : {
        //runTask2
    },
    "task3" : {
        //runTask3
    },
)

This can be enclosed inside a node, in which case they would all share the same workspace or each command can define node inside the task code to run into another node in which case it would have a different workspace for each task.

> **Comments:**
> **username_0**: Hi Michael! I was talking more about downloading files and not uploading. If I did this with downloading, wouldn't it save the files on different nodes rather than a single workspace?  
> **username_2**: Not necessarily, I will give you a better example in a few moments  
> **username_0**: Not sure that matches my case since when I download artifacts, I don't know the exact number of artifacts (using ant-like syntax to download the entire repository)  
> **username_2**: In that case, take a look at https://jenkins.io/doc/pipeline/examples/#jobs-in-parallel You can reuse the same code and just change the loop counter to the number of elements you have to download, and replace the `build job` line with the one to download from artifactory  
> **username_0**: But what if I don't know the number of files I download? I use the double asterisk notation (**) to recursively download all my repository, so neither I know how many there are nor their names.  

---

