<p>My preference is to avoid working with files with the same name as much as possible. However, sometimes this might be unavoidable. Fortunately, file name collisions can be avoided when working with <a href="https://www.nextflow.io/docs/latest/process.html#multiple-input-files" rel="nofollow noreferrer">multiple input files</a> by using the <code>*</code> and <code>?</code> wildcards:</p>
<blockquote>
<p>When a target file name is defined in the input parameter and a collection of files is received by the process, the file name will be appended by a numerical suffix representing its ordinal position in the list.</p>
<p>The target input file name can contain the * and ? wildcards, that can be used to control the name of staged files.</p>
</blockquote>
<p>Note that there is also the <code>stageAs</code> path option (see <a href="https://www.nextflow.io/docs/latest/process.html#input-of-type-path" rel="nofollow noreferrer">input of type 'path'</a>), which can also take a name pattern (using the above wildcards) to let you use a variable in your script. For example:</p>
<pre><code>process samtools_merge {

    conda 'samtools=1.15.1'
    debug true
    stageInMode 'rellink'

    input:
    tuple val(sample), path(bam_files, stageAs: 'mysample*.bam')

    output:
    tuple val(sample), path(&quot;${sample}.bam&quot;)

    &quot;&quot;&quot;
    samtools merge &quot;<span class="math-container">${sample}.bam" $</span>{bam_files}
    ls -nl
    &quot;&quot;&quot;
}

workflow {

    Channel.fromPath( ['./path/to/files/*.bam', './path/to/more_files/*.bam'] ) \
        | map { bam -&gt; tuple( bam.baseName, bam ) } \
        | groupTuple(by: 0, size: 2) \
        | samtools_merge
}
</code></pre>
<p>Results:</p>
<pre><code>N E X T F L O W  ~  version 22.04.0
Launching `script.nf` [elated_montalcini] DSL2 - revision: 1427d64a1d
executor &gt;  local (3)
[83/25654c] process &gt; samtools_merge (3) [100%] 3 of 3 ✔
total 184
lrwxrwxrwx 1 1000 985     34 May 25 12:18 mysample1.bam -&gt; ../../../path/to/files/sample1.bam
lrwxrwxrwx 1 1000 985     39 May 25 12:18 mysample2.bam -&gt; ../../../path/to/more_files/sample1.bam
-rw-r--r-- 1 1000 985 187911 May 25 12:18 sample1.bam

total 128
lrwxrwxrwx 1 1000 985     34 May 25 12:18 mysample1.bam -&gt; ../../../path/to/files/sample2.bam
lrwxrwxrwx 1 1000 985     39 May 25 12:18 mysample2.bam -&gt; ../../../path/to/more_files/sample2.bam
-rw-r--r-- 1 1000 985 129947 May 25 12:18 sample2.bam

total 196
lrwxrwxrwx 1 1000 985     34 May 25 12:18 mysample1.bam -&gt; ../../../path/to/files/sample3.bam
lrwxrwxrwx 1 1000 985     39 May 25 12:18 mysample2.bam -&gt; ../../../path/to/more_files/sample3.bam
-rw-r--r-- 1 1000 985 197225 May 25 12:18 sample3.bam

</code></pre>
<p>If each group contains a variable number of files, you can keep the same input declaration (as above) but you will want use a special <a href="https://github.com/nextflow-io/nextflow/blob/v22.04.3/modules/nextflow/src/main/groovy/nextflow/extension/GroupKey.groovy" rel="nofollow noreferrer">groupKey</a> object so that the collected values can be streamed as soon as possible. It's not well documented, but here's a simple example that sorts the input BAM files prior to merging. Once all of the 'samtools_sort' jobs are finished for a particular sample, that sample can proceed to merging without us having to wait for all the other 'samtools_sort' jobs to be finished (which would otherwise be the case if the 'size' parameter had not been specified):</p>
<pre><code>process samtools_sort {

    tag { bam.baseName }

    conda 'samtools=1.15.1'

    input:
    tuple val(sample), path(bam)

    output:
    tuple val(sample), path(&quot;${bam.baseName}.sorted.bam&quot;)

    &quot;&quot;&quot;
    samtools sort -o &quot;<span class="math-container">${bam.baseName}.sorted.bam" "$</span>{bam}&quot;
    &quot;&quot;&quot;
}

process samtools_merge {

    tag { sample }

    conda 'samtools=1.15.1'

    input:
    tuple val(sample), path(bam_files, stageAs: 'mysample*.bam')

    output:
    tuple val(sample), path(&quot;${sample}.bam&quot;)

    &quot;&quot;&quot;
    samtools merge &quot;<span class="math-container">${sample}.bam" $</span>{bam_files}
    &quot;&quot;&quot;
}
</code></pre>
<pre><code>workflow {

    Channel.fromPath( params.bam_files ) \
        | map { bam -&gt; tuple( bam.baseName, bam ) } \
        | groupTuple() \
        | map { sample, files -&gt; tuple( groupKey(sample, files.size()), files ) } \
        | transpose() \
        | set { bam_files }

    samtools_sort( bam_files ) \
        | groupTuple() \
        | samtools_merge \
        | view()
}
</code></pre>
<p>Results:</p>
<pre><code>$ nextflow run script.nf -ansi-log false --bam_files './path/to/*/*.bam'
N E X T F L O W  ~  version 22.04.0
Launching `script.nf` [cheesy_swanson] DSL2 - revision: da1a82a059
[04/643129] Submitted process &gt; samtools_sort (sample3)
[1a/93a939] Submitted process &gt; samtools_sort (sample1)
[93/fde0a0] Submitted process &gt; samtools_sort (sample3)
[db/208bac] Submitted process &gt; samtools_sort (sample3)
[8f/f2a231] Submitted process &gt; samtools_sort (sample2)
[f3/8f0f9f] Submitted process &gt; samtools_sort (sample2)
[bb/cca23e] Submitted process &gt; samtools_merge (sample1)
[sample1, /home/steve/testing/work/bb/cca23e016a641b201b7e42f9db74ef/sample1.bam]
[ff/bd3a2a] Submitted process &gt; samtools_merge (sample2)
[sample2, /home/steve/testing/work/ff/bd3a2ac83a0691d2b80b04ef97b755/sample2.bam]
[94/a4f8eb] Submitted process &gt; samtools_merge (sample3)
[sample3, /home/steve/testing/work/94/a4f8eb483602e9335b28f5d40058c8/sample3.bam]
</code></pre>
<p>Note that the <a href="https://www.nextflow.io/docs/latest/process.html#input-of-generic-values" rel="nofollow noreferrer">val</a> specified as input to 'samtools_sort' is an object of class <code>nextflow.extension.GroupKey</code>. In the example above, the input to 'samtools_merge' is also a groupKey - but it doesn't need to be. If you need a string value for whatever reason, you can just call '.toString()' to get it back:</p>
<pre><code>    ...

    samtools_sort( bam_files ) \
        | groupTuple() \
        | map { sample_key, files -&gt; tuple( sample_key.toString(), files ) } \
        | samtools_merge \
        | view()
</code></pre>

CC BY-SA 4.0
2022-05-25T14:12:51.580Z
Thank you so much for this very comprehensive answer! I had totally missed that section of the `input` documentation. And as ATpoint notes, this is the second time you help me on a Nextflow question, this has been extremely helpful!
Alexlok
CC BY-SA 4.0
2022-05-25T06:16:47.527Z
2022-05-25T06:16:47.527Z
Steve
<p>I have a process that emits BAM files with a key containing the sample name:</p>
<pre><code>process source{
  output:
    tuple val(sample), path(&quot;sample.bam&quot;) into ch

  [...]

}
</code></pre>
<p>1st question: I have several files per <code>sample</code>, let's assume 2 for now. I want to merge them together with <a href="http://www.htslib.org/doc/samtools-merge.html" rel="nofollow noreferrer"><code>samtools merge</code></a>. I can regroup that channel with <a href="https://www.nextflow.io/docs/latest/operator.html#grouptuple" rel="nofollow noreferrer"><code>groupTuple()</code></a>, but how can I recover them in the next process? Considering they all have the same name:</p>
<pre><code>grouped_ch = ch.groupTuple(by: 0, size: 2)

process merge{
  input:
    grouped_ch

  shell:
  '''
  samtool merge sample.bam sample.bam &gt; !{sample}.bam
  '''
}
</code></pre>
<p>Is there perhaps a way in the <code>input:</code> block to say that I expect a list, and name the list components?</p>
<p>2nd question: In practice, I have a variable number of &quot;sample.bam&quot; files for each <code>sample</code> value, does that change the way to define the <code>input:</code>? It also appears I will need to use the built-in <code>groupKey()</code>, is there documentation available? I have read <a href="https://github.com/nextflow-io/nextflow/issues/796#issuecomment-407108667" rel="nofollow noreferrer">this Github issue</a>, but it would help if a general description was available.</p>

<nextflow>
recover files after groupTuple
# recover files after groupTuple

(Asked by: username_0 on 2022-05-24)

I have a process that emits BAM files with a key containing the sample name:
process source{
  output:
    tuple val(sample), path("sample.bam") into ch

  [...]

}

1st question: I have several files per sample, let's assume 2 for now. I want to merge them together with samtools merge. I can regroup that channel with groupTuple(), but how can I recover them in the next process? Considering they all have the same name:
grouped_ch = ch.groupTuple(by: 0, size: 2)

process merge{
  input:
    grouped_ch

  shell:
  '''
  samtool merge sample.bam sample.bam > !{sample}.bam
  '''
}

Is there perhaps a way in the input: block to say that I expect a list, and name the list components?
2nd question: In practice, I have a variable number of "sample.bam" files for each sample value, does that change the way to define the input:? It also appears I will need to use the built-in groupKey(), is there documentation available? I have read this Github issue, but it would help if a general description was available.

**Tags:** nextflow

---

## 1 answers

### Answer by username_1 on 2022-05-25. Score: 3 | Accepted answer

My preference is to avoid working with files with the same name as much as possible. However, sometimes this might be unavoidable. Fortunately, file name collisions can be avoided when working with multiple input files by using the * and ? wildcards:

When a target file name is defined in the input parameter and a collection of files is received by the process, the file name will be appended by a numerical suffix representing its ordinal position in the list.
The target input file name can contain the * and ? wildcards, that can be used to control the name of staged files.

Note that there is also the stageAs path option (see input of type 'path'), which can also take a name pattern (using the above wildcards) to let you use a variable in your script. For example:
process samtools_merge {

    conda 'samtools=1.15.1'
    debug true
    stageInMode 'rellink'

    input:
    tuple val(sample), path(bam_files, stageAs: 'mysample*.bam')

    output:
    tuple val(sample), path("${sample}.bam")

    """
    samtools merge "${sample}.bam" ${bam_files}
    ls -nl
    """
}

workflow {

    Channel.fromPath( ['./path/to/files/*.bam', './path/to/more_files/*.bam'] ) \
        | map { bam -> tuple( bam.baseName, bam ) } \
        | groupTuple(by: 0, size: 2) \
        | samtools_merge
}

Results:
N E X T F L O W  ~  version 22.04.0
Launching `script.nf` [elated_montalcini] DSL2 - revision: 1427d64a1d
executor >  local (3)
[83/25654c] process > samtools_merge (3) [100%] 3 of 3 ✔
total 184
lrwxrwxrwx 1 1000 985     34 May 25 12:18 mysample1.bam -> ../../../path/to/files/sample1.bam
lrwxrwxrwx 1 1000 985     39 May 25 12:18 mysample2.bam -> ../../../path/to/more_files/sample1.bam
-rw-r--r-- 1 1000 985 187911 May 25 12:18 sample1.bam

total 128
lrwxrwxrwx 1 1000 985     34 May 25 12:18 mysample1.bam -> ../../../path/to/files/sample2.bam
lrwxrwxrwx 1 1000 985     39 May 25 12:18 mysample2.bam -> ../../../path/to/more_files/sample2.bam
-rw-r--r-- 1 1000 985 129947 May 25 12:18 sample2.bam

total 196
lrwxrwxrwx 1 1000 985     34 May 25 12:18 mysample1.bam -> ../../../path/to/files/sample3.bam
lrwxrwxrwx 1 1000 985     39 May 25 12:18 mysample2.bam -> ../../../path/to/more_files/sample3.bam
-rw-r--r-- 1 1000 985 197225 May 25 12:18 sample3.bam


If each group contains a variable number of files, you can keep the same input declaration (as above) but you will want use a special groupKey object so that the collected values can be streamed as soon as possible. It's not well documented, but here's a simple example that sorts the input BAM files prior to merging. Once all of the 'samtools_sort' jobs are finished for a particular sample, that sample can proceed to merging without us having to wait for all the other 'samtools_sort' jobs to be finished (which would otherwise be the case if the 'size' parameter had not been specified):
process samtools_sort {

    tag { bam.baseName }

    conda 'samtools=1.15.1'

    input:
    tuple val(sample), path(bam)

    output:
    tuple val(sample), path("${bam.baseName}.sorted.bam")

    """
    samtools sort -o "${bam.baseName}.sorted.bam" "${bam}"
    """
}

process samtools_merge {

    tag { sample }

    conda 'samtools=1.15.1'

    input:
    tuple val(sample), path(bam_files, stageAs: 'mysample*.bam')

    output:
    tuple val(sample), path("${sample}.bam")

    """
    samtools merge "${sample}.bam" ${bam_files}
    """
}

workflow {

    Channel.fromPath( params.bam_files ) \
        | map { bam -> tuple( bam.baseName, bam ) } \
        | groupTuple() \
        | map { sample, files -> tuple( groupKey(sample, files.size()), files ) } \
        | transpose() \
        | set { bam_files }

    samtools_sort( bam_files ) \
        | groupTuple() \
        | samtools_merge \
        | view()
}

Results:
$ nextflow run script.nf -ansi-log false --bam_files './path/to/*/*.bam'
N E X T F L O W  ~  version 22.04.0
Launching `script.nf` [cheesy_swanson] DSL2 - revision: da1a82a059
[04/643129] Submitted process > samtools_sort (sample3)
[1a/93a939] Submitted process > samtools_sort (sample1)
[93/fde0a0] Submitted process > samtools_sort (sample3)
[db/208bac] Submitted process > samtools_sort (sample3)
[8f/f2a231] Submitted process > samtools_sort (sample2)
[f3/8f0f9f] Submitted process > samtools_sort (sample2)
[bb/cca23e] Submitted process > samtools_merge (sample1)
[sample1, /home/steve/testing/work/bb/cca23e016a641b201b7e42f9db74ef/sample1.bam]
[ff/bd3a2a] Submitted process > samtools_merge (sample2)
[sample2, /home/steve/testing/work/ff/bd3a2ac83a0691d2b80b04ef97b755/sample2.bam]
[94/a4f8eb] Submitted process > samtools_merge (sample3)
[sample3, /home/steve/testing/work/94/a4f8eb483602e9335b28f5d40058c8/sample3.bam]

Note that the val specified as input to 'samtools_sort' is an object of class nextflow.extension.GroupKey. In the example above, the input to 'samtools_merge' is also a groupKey - but it doesn't need to be. If you need a string value for whatever reason, you can just call '.toString()' to get it back:
    ...

    samtools_sort( bam_files ) \
        | groupTuple() \
        | map { sample_key, files -> tuple( sample_key.toString(), files ) } \
        | samtools_merge \
        | view()

> **Comments:**
> **username_0**: Thank you so much for this very comprehensive answer! I had totally missed that section of the `input` documentation. And as ATpoint notes, this is the second time you help me on a Nextflow question, this has been extremely helpful!  

---

