<p>A better way would be to stage the files in a sub-directory, which would allow you to simply pass in a glob pattern to your tool. This can be achieved using the <code>path</code> input type and a string containing the <code>*</code> or <code>?</code> wildcards to control the names of staged files<sup><a href="https://www.nextflow.io/docs/stable/process.html#multiple-input-files" rel="noreferrer">1</a></sup>. Here, the files are named as the source files in a sub-directory called 'staged':</p>
<pre><code>process PROCESS_B{

    input:
    path 'staged/*'

    script:
    &quot;&quot;&quot;
    mytool staged/*
    &quot;&quot;&quot;
}
</code></pre>
<p>Note that you can also use the <code>stageAs</code> option for this, if a variable is somehow useful:</p>
<pre><code>process PROCESS_B{

    input:
    path all_files_list, stageAs: 'staged/*'

    script:
    &quot;&quot;&quot;
    echo &quot;Staged ${all_files_list.size()} files...&quot;
    mytool staged/*
    &quot;&quot;&quot;
}
</code></pre>

CC BY-SA 4.0
2024-09-27T17:14:33.853Z
Oh man, I can't believe I missed this in the docs. This is very elegant. Thank you!
DLJ
CC BY-SA 4.0
2024-09-27T14:17:41.497Z
2024-09-27T14:17:41.497Z
Steve
<p>Consider a hypothetical pipeline shown below, where many files need to be collected at one stage in the pipeline:</p>
<p><img src="https://i.sstatic.net/Lhfs08Nd.png" alt="Hypothetical Pipeline" /></p>
<p>It seems like the idiomatic way of accomplishing this is to pass <code>Process B</code> a list of filenames, which then passes those file names to the tool in question:</p>
<pre><code>workflow {
    channel_with_many_files = channel.of(...)

    PROCESS_A(channel_with_many_files)
    
    all_files_together = PROCESS_A.out.collect()

    PROCESS_B(all_files_together)
}
</code></pre>
<p>And then in the process B definition:</p>
<pre><code>process PROCESS_B{
    input:
    path(all_files_list)

    output:
    path(single_output_file)
    
    script:

    &quot;&quot;&quot;
    ./mytool ${all_files_list}
    &quot;&quot;&quot;
}
</code></pre>
<p>The problem with this, in my opinion, is that when <code>n</code> (number of files) grows to be very large (say &gt; 2000), the command that is generated in <code>PROCESS_B</code> is unweidly and a bit messy.</p>
<p>Is this the way to deal with such a pattern in Nextflow, or is there a better way?</p>

<nextflow>
Nextflow Best Practice When Collecting a Large Number of Files in One Step
# Nextflow Best Practice When Collecting a Large Number of Files in One Step

(Asked by: username_0 on 2024-09-27)

Consider a hypothetical pipeline shown below, where many files need to be collected at one stage in the pipeline:

It seems like the idiomatic way of accomplishing this is to pass Process B a list of filenames, which then passes those file names to the tool in question:
workflow {
    channel_with_many_files = channel.of(...)

    PROCESS_A(channel_with_many_files)
    
    all_files_together = PROCESS_A.out.collect()

    PROCESS_B(all_files_together)
}

And then in the process B definition:
process PROCESS_B{
    input:
    path(all_files_list)

    output:
    path(single_output_file)
    
    script:

    """
    ./mytool ${all_files_list}
    """
}

The problem with this, in my opinion, is that when n (number of files) grows to be very large (say > 2000), the command that is generated in PROCESS_B is unweidly and a bit messy.
Is this the way to deal with such a pattern in Nextflow, or is there a better way?

**Tags:** nextflow

---

## 1 answers

### Answer by username_1 on 2024-09-27. Score: 5 | Accepted answer

A better way would be to stage the files in a sub-directory, which would allow you to simply pass in a glob pattern to your tool. This can be achieved using the path input type and a string containing the * or ? wildcards to control the names of staged files1. Here, the files are named as the source files in a sub-directory called 'staged':
process PROCESS_B{

    input:
    path 'staged/*'

    script:
    """
    mytool staged/*
    """
}

Note that you can also use the stageAs option for this, if a variable is somehow useful:
process PROCESS_B{

    input:
    path all_files_list, stageAs: 'staged/*'

    script:
    """
    echo "Staged ${all_files_list.size()} files..."
    mytool staged/*
    """
}

> **Comments:**
> **username_0**: Oh man, I can't believe I missed this in the docs. This is very elegant. Thank you!  

---

