<p>I hope this helps. We had a similar issue and solved it using the below method.</p>

<p>Our project contains about 12 or so different components (programs) that we wanted to build in parallel. Because some take longer than others, we decided to break them up into 4 individual "tracks" that would run in parallel. To begin, assume we have an array where each element is a new component object. The object has some properties, like name for the name of the component and track for which track, 1, 2, 3, or 4, that the component is assigned to. I'm not focusing on this part as it's probably not relevant to what you're trying to do. </p>

<p>First we define our <code>createParallelStage</code> function which iterates over an ArrayList of components, setting them up to build.</p>

<pre><code>def createParallelStage(ArrayList components) {
    return{
        // iterate over list of components objects
        components.each { component -&gt;
            dir(buildFolder) {
                // build the component
                sh "./build.sh ${component}"

            }
        }
    }
}
</code></pre>

<p>We'll use a list of component objects called BuildComponents, iterate over them, and add them to <code>parallelGroups[i]</code> where <code>i</code> is the value of the objects <code>track</code> property</p>

<pre><code>BuildComponents.each { component -&gt;
    parallelGroups[component.track].add(component.name)
    echo "${component.name} is going to parallel track ${component.track}"
    // example: IPCService is going to parallel track 3
    }
}
</code></pre>

<p>We're now left with iterating over <code>parallelGroups</code>, passing each element to <code>createParallelStage()</code> while adding it to the <code>parallelStages</code> ArrayList.</p>

<pre><code>parallelGroups.each{i, component -&gt;
    parallelStages.put(i, createParallelStage(component))
}
</code></pre>

<p>We then set <code>failFast</code> to true, so if any of our components fail, the whole build job will fail, and then pass our list of <code>parallelStages</code> to the <code>parallel()</code> function.</p>

<pre><code>parallelStages.failFast = true
parallel(parallelStages)
</code></pre>

<p>I hope this was specific to the question you were asking. I wish I could be a little more detailed, but then it might be too specific of a solution to work for you.</p>

CC BY-SA 4.0
2019-12-31T15:26:27.783Z
2019-12-31T15:26:27.783Z
Argyle
<p>This is quite complicated to achieve in Jenkins. We had a similar issue and here's how we solved it:</p>

<ol>
<li>We have a shared library where we keep our scripts ins <code>./vars/someScriptName.groovy</code></li>
<li>We created a <code>groovy</code> script that generates the stages.</li>
</ol>

<p>Here's a working pipeline code for you to try it out:</p>

<pre><code>def generateITParallelStages(body)
{
    def config = [:]
    config.stages = []

    body.resolveStrategy = Closure.DELEGATE_FIRST
    body.delegate = config
    body()

    def buildStages = [:]
    config.stages.each { String stage, closure -&gt;
        // default settings, if you want to have any
        def settings = [:]
        settings.type = 'online'; // if you don't define the type, it will be 'online' by default
        settings.suite = 'internal'; // if you don't define the suite it will be 'internal' by default

        closure.resolveStrategy = Closure.DELEGATE_FIRST
        closure.delegate = settings
        closure()

        buildStages.put(stage, prepareStage(stage, settings))
    }

    return buildStages
}

def prepareStage(String stage, Map settings)
{
    return {
        print "Run ${settings.type} application  '\\n${stage}' in test suite '${settings.suite}'"
    }
}

pipeline {
    agent any
    stages {
        stage('Add regression tests') {
            steps {
                script {
                    parallel generateITParallelStages {
                        stages = [
                                webgoat: {
                                    type = 'offline'
                                    suite = 'open-source'
                                },
                                'juice-shop': {
                                    type = 'offline'
                                    suite = 'open-source'
                                },
                                'our-website': {
                                    type = 'online'
                                    suite = 'internal'
                                }
                        ]
                    }
                }
            }
        }
    }
}
</code></pre>

<p>And here's the result:</p>

<p><a href="https://i.sstatic.net/am8XK.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/am8XK.png" alt="pipeline-view"></a></p>

<p>This is the only way we have been able to dynamically generate the parallel stages. Everything else will fail (at least at the time of writing this). </p>

<p>Other people seem to be having a hard time figuring this out as well - <a href="https://issues.jenkins-ci.org/browse/JENKINS-53032" rel="nofollow noreferrer">JENKINS-53032</a>. If you feel the same make some noise :)</p>

CC BY-SA 4.0
2020-01-12T23:15:34.153Z
I would like to set a docker agent on the stage, but I am getting an error that seems to indicate I can only use step interface classes as operations in the stage execution. Where would I define this setting? I have tried to find a JavaDoc that shows the underlying classes of the API, but have come up empty.
Jörn Guy Süß
CC BY-SA 4.0
2020-01-12T23:19:29.950Z
You would need to set the `Add regression tests` to run on `none` agent and then have in the `prepareStage` a `node(settings.agent) { ... }` which starts the parallel stage.
tftd
CC BY-SA 4.0
2020-01-13T03:24:41.763Z
Sorry for being such a nuisance. Is there a way I can add a sequence to the stage? I have tried calling the `stages {}` closure, but it is not valid. I assume I am in a scripted environment now, guessing from your previous instruction, which works great!
Jörn Guy Süß
CC BY-SA 4.0
2020-01-13T13:44:26.940Z
That's correct - while in `prepareStage` you are writing `scripted` pipeline code. This means, there is no `post {}`, no `stages {}` and other "sugar" from the declarative pipes. I assume you want to add a few stages on top of the running parallel stage (i.e. have `unit-test`, `integration-test` in `juice-shop`). To do that, just have `node(settings.agent) { stage('ut') {...} stage('it') {...} }`. Checkout https://jenkins.io/doc/book/pipeline/#scripted-pipeline-fundamentals. If that's not what you want - please open up a new question and I'll try to answer.
tftd
CC BY-SA 4.0
2019-12-27T16:09:42.330Z
2019-12-27T18:00:21.757Z
2019-12-27T18:00:21.757Z
tftd
<p>In declarative pipelines, Jenkins allows the definition of <a href="https://jenkins.io/doc/book/pipeline/syntax/#parallel" rel="noreferrer">parallel stages</a>. It further allows <a href="https://jenkins.io/doc/book/pipeline/syntax/#script" rel="noreferrer">scripted pipeline general purpose scripts</a> to create and manipulate the artifacts of the declarative pipeline. For example, it can create <code>stages</code> dynamically.</p>

<p>I would like to know how I can <em>dynamically</em> create <em>parallel executing stages</em>. Please note I am not talking about <em>steps</em> or actions, which are solved here:</p>

<p><a href="https://devops.stackexchange.com/questions/3073/how-to-properly-achieve-dynamic-parallel-action-with-a-declarative-pipeline">How to properly achieve dynamic parallel action with a declarative pipeline?</a></p>

<p>Below is a simple Jenkinsfile that dynamically adds stages using groovy code. These stages are sequential. I would like a Jenkinsfile that adds stages dynamically as below, but uses the <code>parallel</code> construct at the <code>stage</code> level. As a result, three stages that run parallel should be generated by the program.</p>

<pre class="lang-java prettyprint-override"><code>pipeline {
    agent any
    stages {
        stage('Add regression tests') {
            steps {
                addStage('offline','open-source','webgoat')
                addStage('offline','open-source','juice-shop')
                addStage('online','internal','our-website')
            }
        }
    }
}

final void addStage(final String type, final String suite,final String application) {
    script {
        stage("Run ${type} application  '\n${application}' in test suite '${suite}'") {

            runTestOnApplication(type,suite,application)

        }
    }
}

final void runTestOnApplication(final String type, final String suite,final String application) {
    labelledShell label: "Run test shell script for ${type} application '${application}' in test suite '${suite}'", script: 'test.bash'
}

</code></pre>

CC BY-SA 4.0
2019-12-27T16:25:55.490Z
@tftd Do you know whether there is a related Jira Jenkins issue for this? If not, perhaps you could create it and link it to this Q&A. Perhaps creating an issue on the Jenkins Jira board could create awareness to make it less complicated?
030
CC BY-SA 4.0
2019-12-27T16:34:17.663Z
@030 It looks like [JENKINS-53032](https://issues.jenkins-ci.org/browse/JENKINS-53032) asking for something similar.
tftd
CC BY-SA 4.0
2019-12-27T16:37:48.477Z
@tftd Well done. Perhaps you could add a link to this ticket to your answer and a link to your answer to the jenkins ticket?
030
<jenkins-pipeline><jenkinsfile>
How to define dynamic parallel stages in a Jenkinsfile?
Jörn Guy Süß
# How to define dynamic parallel stages in a Jenkinsfile?

(Asked by: username_0 on 2019-11-22)

In declarative pipelines, Jenkins allows the definition of parallel stages. It further allows scripted pipeline general purpose scripts to create and manipulate the artifacts of the declarative pipeline. For example, it can create stages dynamically.
I would like to know how I can dynamically create parallel executing stages. Please note I am not talking about steps or actions, which are solved here:
How to properly achieve dynamic parallel action with a declarative pipeline?
Below is a simple Jenkinsfile that dynamically adds stages using groovy code. These stages are sequential. I would like a Jenkinsfile that adds stages dynamically as below, but uses the parallel construct at the stage level. As a result, three stages that run parallel should be generated by the program.
pipeline {
    agent any
    stages {
        stage('Add regression tests') {
            steps {
                addStage('offline','open-source','webgoat')
                addStage('offline','open-source','juice-shop')
                addStage('online','internal','our-website')
            }
        }
    }
}

final void addStage(final String type, final String suite,final String application) {
    script {
        stage("Run ${type} application  '\n${application}' in test suite '${suite}'") {

            runTestOnApplication(type,suite,application)

        }
    }
}

final void runTestOnApplication(final String type, final String suite,final String application) {
    labelledShell label: "Run test shell script for ${type} application '${application}' in test suite '${suite}'", script: 'test.bash'
}

**Tags:** jenkins-pipeline, jenkinsfile

> **Comments:**
> **username_1**: @username_2 Do you know whether there is a related Jira Jenkins issue for this? If not, perhaps you could create it and link it to this Q&A. Perhaps creating an issue on the Jenkins Jira board could create awareness to make it less complicated?  
> **username_2**: @username_1 It looks like [JENKINS-53032](https://issues.jenkins-ci.org/browse/JENKINS-53032) asking for something similar.  
> **username_1**: @username_2 Well done. Perhaps you could add a link to this ticket to your answer and a link to your answer to the jenkins ticket?  

---

## 2 answers

### Answer by username_2 on 2019-12-27. Score: 3 | Accepted answer

This is quite complicated to achieve in Jenkins. We had a similar issue and here's how we solved it:

We have a shared library where we keep our scripts ins ./vars/someScriptName.groovy
We created a groovy script that generates the stages.

Here's a working pipeline code for you to try it out:
def generateITParallelStages(body)
{
    def config = [:]
    config.stages = []

    body.resolveStrategy = Closure.DELEGATE_FIRST
    body.delegate = config
    body()

    def buildStages = [:]
    config.stages.each { String stage, closure ->
        // default settings, if you want to have any
        def settings = [:]
        settings.type = 'online'; // if you don't define the type, it will be 'online' by default
        settings.suite = 'internal'; // if you don't define the suite it will be 'internal' by default

        closure.resolveStrategy = Closure.DELEGATE_FIRST
        closure.delegate = settings
        closure()

        buildStages.put(stage, prepareStage(stage, settings))
    }

    return buildStages
}

def prepareStage(String stage, Map settings)
{
    return {
        print "Run ${settings.type} application  '\\n${stage}' in test suite '${settings.suite}'"
    }
}

pipeline {
    agent any
    stages {
        stage('Add regression tests') {
            steps {
                script {
                    parallel generateITParallelStages {
                        stages = [
                                webgoat: {
                                    type = 'offline'
                                    suite = 'open-source'
                                },
                                'juice-shop': {
                                    type = 'offline'
                                    suite = 'open-source'
                                },
                                'our-website': {
                                    type = 'online'
                                    suite = 'internal'
                                }
                        ]
                    }
                }
            }
        }
    }
}

And here's the result:

This is the only way we have been able to dynamically generate the parallel stages. Everything else will fail (at least at the time of writing this). 
Other people seem to be having a hard time figuring this out as well - JENKINS-53032. If you feel the same make some noise :)

> **Comments:**
> **username_0**: I would like to set a docker agent on the stage, but I am getting an error that seems to indicate I can only use step interface classes as operations in the stage execution. Where would I define this setting? I have tried to find a JavaDoc that shows the underlying classes of the API, but have come up empty.  
> **username_2**: You would need to set the `Add regression tests` to run on `none` agent and then have in the `prepareStage` a `node(settings.agent) { ... }` which starts the parallel stage.  
> **username_0**: Sorry for being such a nuisance. Is there a way I can add a sequence to the stage? I have tried calling the `stages {}` closure, but it is not valid. I assume I am in a scripted environment now, guessing from your previous instruction, which works great!  
> **username_2**: That's correct - while in `prepareStage` you are writing `scripted` pipeline code. This means, there is no `post {}`, no `stages {}` and other "sugar" from the declarative pipes. I assume you want to add a few stages on top of the running parallel stage (i.e. have `unit-test`, `integration-test` in `juice-shop`). To do that, just have `node(settings.agent) { stage('ut') {...} stage('it') {...} }`. Checkout https://jenkins.io/doc/book/pipeline/#scripted-pipeline-fundamentals. If that's not what you want - please open up a new question and I'll try to answer.  

---

### Answer by username_3 on 2019-12-31. Score: 1

I hope this helps. We had a similar issue and solved it using the below method.
Our project contains about 12 or so different components (programs) that we wanted to build in parallel. Because some take longer than others, we decided to break them up into 4 individual "tracks" that would run in parallel. To begin, assume we have an array where each element is a new component object. The object has some properties, like name for the name of the component and track for which track, 1, 2, 3, or 4, that the component is assigned to. I'm not focusing on this part as it's probably not relevant to what you're trying to do. 
First we define our createParallelStage function which iterates over an ArrayList of components, setting them up to build.
def createParallelStage(ArrayList components) {
    return{
        // iterate over list of components objects
        components.each { component ->
            dir(buildFolder) {
                // build the component
                sh "./build.sh ${component}"

            }
        }
    }
}

We'll use a list of component objects called BuildComponents, iterate over them, and add them to parallelGroups[i] where i is the value of the objects track property
BuildComponents.each { component ->
    parallelGroups[component.track].add(component.name)
    echo "${component.name} is going to parallel track ${component.track}"
    // example: IPCService is going to parallel track 3
    }
}

We're now left with iterating over parallelGroups, passing each element to createParallelStage() while adding it to the parallelStages ArrayList.
parallelGroups.each{i, component ->
    parallelStages.put(i, createParallelStage(component))
}

We then set failFast to true, so if any of our components fail, the whole build job will fail, and then pass our list of parallelStages to the parallel() function.
parallelStages.failFast = true
parallel(parallelStages)

I hope this was specific to the question you were asking. I wish I could be a little more detailed, but then it might be too specific of a solution to work for you.


---

