<p>I think you are on the right track using <code>groupTuple(sort: true)</code>. Another way to get what you want would be to instead parse the <code>fromFilePairs</code> grouping key. Unless you're looking to do more than just concatenate (and decompress) FASTQ files, I think all the process really requires is an output prefix (in addition to the FASTQ files of course), for example:</p>
<pre><code>params.reads = &quot;/sc/arion/MM-*_R{1,2}_001.fastq.gz&quot;


process concat_reads {

    tag { prefix }

    input:
    tuple val(prefix), path(reads, stageAs: 'reads/*')

    output:
    tuple val(prefix), path(&quot;${prefix}.fastq&quot;)

    &quot;&quot;&quot;
    zcat -f <span class="math-container">${reads} &gt; "$</span>{prefix}.fastq&quot;
    &quot;&quot;&quot;
}
</code></pre>
<pre><code>workflow {

    Channel.fromFilePairs( params.reads ) \
        | branch { key, fastqs -&gt;

            def matcher = key =~ /^(MM-\w+)-DNA-(\w+)-.*_L(\d+).*$/

            matches: matcher
                def (_, sample_id, sample_type, lane) = matcher[0].findAll()

                def new_sample_id = &quot;<span class="math-container">${sample_id}_$</span>{sample_type}&quot;

                return tuple( new_sample_id, lane as int, *fastqs )

            other: true
                return tuple( key, *fastqs )
        } \
        | set { samples }

    samples.matches \
        | groupTuple( sort: true ) \
        | flatMap { sample_id, lanes, fq1s, fq2s -&gt;
            [
                tuple( &quot;<span class="math-container">${sample_id}_R1", fq1s ),
                tuple( "$</span>{sample_id}_R2&quot;, fq2s ),
            ]
        } \
        | concat_reads \
        | branch { prefix, fastq -&gt;

            def idx = prefix.lastIndexOf('_')

            def sample_id = prefix.substring(0, idx)
            def suffix = prefix.substring(idx + 1)

            read1: suffix == 'R1'
                return tuple( sample_id, fastq )
            read2: suffix == 'R2'
                return tuple( sample_id, fastq )
        } \
        | set { result }

    result.read1 \
        | join( result.read2 ) \
        | map { sample_id, fastq1, fastq2 -&gt;

            tuple( sample_id, [ fastq1, fastq2 ] )
        } \
        | view()
}
</code></pre>
<p>Results:</p>
<pre><code>$ nextflow run main.nf
N E X T F L O W  ~  version 23.04.1
Launching `main.nf` [jolly_noether] DSL2 - revision: 4a81f7a875
executor &gt;  local (8)
[b8/a0b534] process &gt; concat_reads (MM-265_T_R2) [100%] 8 of 8 ✔
[MM-465_T, [/path/to/work/9c/8b6fd328c848379423a0aba098cb74/MM-465_T_R1.fastq, /path/to/work/34/913780b1da902b5035f3bf56aac038/MM-465_T_R2.fastq]]
[MM-465_N, [/path/to/work/69/595120a1ee0a42217d06855ca9d1bc/MM-465_N_R1.fastq, /path/to/work/8f/31c406c5180073de577502fdb88fbf/MM-465_N_R2.fastq]]
[MM-265_N, [/path/to/work/1b/9c17cefb5aa51b3ee69832dc65d915/MM-265_N_R1.fastq, /path/to/work/d0/aac9f46e4a7f7165693b2537ad5630/MM-265_N_R2.fastq]]
[MM-265_T, [/path/to/work/d5/6295ff109df3ccb829a6f0a97b6e45/MM-265_T_R1.fastq, /path/to/work/b8/a0b53499ca13a523d811ecf772449b/MM-265_T_R2.fastq]]
</code></pre>
<p>Note that unless the FASTQ files are small, I would try to avoid decompressing the FASTQ files if possible. Most bioinformatics tools that deal with large FASTQ files will likely have gzip support, meaning you can replace the <code>zcat -f</code> with a <code>cat</code> in the example above (and remember to add the <code>.gz</code> extension to the output filename).</p>

CC BY-SA 4.0
2023-07-20T11:42:12.803Z
Thank you for your reply and code. I'm sorry, I cannot understand what you did in the code as it is scary and complicated for me. May I know what is wrong in my code? I've .fastq extension as a suffix. May be a little explanation of the steps you did would be highly helpful to me. For example, when to `branch` branch.  Or, how do I put `normal` in the concatenated files?
Death Metal
CC BY-SA 4.0
2023-07-20T11:53:37.107Z
what did you do after `concat_read  \ | branch` what is this doing? Why do we need this?
Death Metal
CC BY-SA 4.0
2023-07-20T12:16:14.100Z
Hi I've edited my code for output by providing `tuple val` and `path` in parenthesis. However I get same error. The error messages don't convey any meaning, they are incredibly cryptic. :(
Death Metal
CC BY-SA 4.0
2023-07-20T12:55:36.067Z
@DeathMetal I suspect your error message is being truncated because it tells us exactly why it failed: `.command.sh: line 2: null: command not found`. Try running your workflow instead using `nextflow run main.nf -ansi-log false`. Since your script block returns `null`, this becomes the command Nextflow tries to run for better or worse. I think you could replace this with a script block containing: `zcat -f ${datasetFile} > "${output_file_name}"` to get it running. Also, use `normal_reads.map{ key, files ->` to unpack the `fromFilePairs` output.
Steve
CC BY-SA 4.0
2023-07-20T13:06:02.523Z
@DeathMetal Note that processes do not necessarily _process_ items in the order that they are received. For example, `concat_reads` can return results in any order. So, if we want to get back the reads where the R1 and R2 files are the first and second in the pair, respectively, then we need some way to separate the outputs so they can be joined correctly. In the above, I just used [`branch`](https://www.nextflow.io/docs/latest/operator.html#branch) to create two channels: `read1` and `read2` and then a [`join`](https://www.nextflow.io/docs/latest/operator.html#join) to put them back together.
Steve
CC BY-SA 4.0
2023-07-20T13:15:12.783Z
@DeathMetal Also, if you wanted to replace `N` with `normal` and `T` with `tumor`, I think the place to do this would when defining the `new_sample_id` in the above. You could use a `Map` for example.
Steve
CC BY-SA 4.0
2023-07-20T13:57:37.903Z
Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/147354/discussion-between-death-metal-and-steve).
Death Metal
CC BY-SA 4.0
2023-07-20T19:02:21.790Z
I am unable to understand `def (_, sample_id, sample_type, lane) = matcher[0].findAll()` why there is a `_` Is there a way to print/echo these outputs?
Death Metal
CC BY-SA 4.0
2023-07-20T20:25:55.397Z
I want output concatenated from  `concat_reads` process only. They are generated in the folder as required. Is it necessary to write/do code below it where you `join`?
Death Metal
CC BY-SA 4.0
2023-07-21T00:16:26.790Z
@DeathMetal If you just need the output of `concat_reads`, the code after and including the last `branch` statement is not necessary. But usually you would want to do something with these new pairs of files, which is why I included some code to show how this could be done. In reality, there are lots of ways to do this though. The underscore is just a placeholder for the full match. In Python and Groovy multi-assignment, at least, it is convention is to use an underscore for un-used variables. Use `samples.matches.view()` or insert a `println(your_variable)` into the `branch` statement to debug
Steve
CC BY-SA 4.0
2023-07-20T05:53:46.630Z
2023-07-20T05:53:46.630Z
Steve
<p>I'd like to merge files from different lanes for forward and reverse reads. For e.g., if a sample A has three lanes in forward, and three reverse, ultimately one reverse and one forward is to be generated.  I have normal and tumor for each sample. N as normal and T as tumor.<br />
The lanes files are for e.g. as:</p>
<pre><code>MM-265-DNA-N-01-01_L003_R1_001.fastq.gz
MM-265-DNA-N-01-01_L003_R2_001.fastq.gz
MM-265-DNA-N-01-01_L004_R1_001.fastq.gz
MM-265-DNA-N-01-01_L004_R2_001.fastq.gz

MM-265-DNA-T-01-01_L003_R1_001.fastq.gz
MM-265-DNA-T-01-01_L003_R2_001.fastq.gz
MM-265-DNA-T-01-01_L004_R1_001.fastq.gz
MM-265-DNA-T-01-01_L004_R2_001.fastq.gz

MM-465-DNA-N-01-01_L003_R1_001.fastq.gz
MM-465-DNA-N-01-01_L003_R2_001.fastq.gz
MM-465-DNA-N-01-01_L004_R1_001.fastq.gz
MM-465-DNA-N-01-01_L004_R2_001.fastq.gz

MM-465-DNA-T-01-01_L003_R1_001.fastq.gz
MM-465-DNA-T-01-01_L003_R2_001.fastq.gz
MM-465-DNA-T-01-01_L004_R1_001.fastq.gz
MM-465-DNA-T-01-01_L004_R2_001.fastq.gz  
</code></pre>
<p>The final output needs to be as:</p>
<pre><code>MM-465_normal_R1.fastq MM-465_normal_R2.fastq
MM-465_tumor_R1.fastq MM-465_tumor_R2.fastq
MM-265_normal_R1.fastq MM-265_normal_R2.fastq
MM-265_tumor_R1.fastq MM-265_tumor_R2.fastq
</code></pre>
<p>I've code below broken in three parts:</p>
<ol>
<li>regular expression</li>
<li>process that iterates over files</li>
<li>workflow that calls the above two code blocks</li>
</ol>
<p>regex:</p>
<pre><code>def get_sample_id( file ) {
def matcher = (file =~ /(MM-\d+)-DNA-N-.+/)
    if (matcher) {
        return matcher[0][1]
    } else {
        return ''
    }
}
</code></pre>
<p>process:</p>
<pre><code>process file_print {

debug true

input:

tuple val(sample_id), path(datasetFile)
val(sample_status) 
val(read_value)

output:
publishDir '/arion/learn_new_concepts/', mode:'copy'

tuple val (sample_id), path(&quot;${output_file_name}&quot;)

script:

output_file_name = &quot;<span class="math-container">${sample_id}_$</span>{sample_status}_${read_value}.fastq&quot;

for (g in datasetFile){
println(g)

&quot;zcat <span class="math-container">${g} &gt;&gt; $</span>{output_file_name} &quot;

}

}
</code></pre>
<p>Workflow:</p>
<pre><code>workflow {

normal_reads=Channel.fromFilePairs(&quot;/sc/arion/MM-*-N-*_L00*_R{1,2}_*.fastq.gz&quot;)
//MM-2658-DNA-N-01-01_L004_R2_001.fastq.gz  //

normal_forward_reads=normal_reads
        .map{ file -&gt; return tuple( get_sample_id(file), file[1][0] )}.groupTuple(sort:true ) 

file_print(normal_forward_reads,&quot;normal&quot;,&quot;R1&quot;).view() /// similar for reverse [1][1], R2 .. for tumor pass reads, &quot;tumor&quot;, &quot;R1&quot;/R2&quot;

}
</code></pre>
<p>I have not put code for reverse, but I get error as:</p>
<pre><code>Caused by:   Process `file_print (1)` terminated with an error exit status (127) Command executed:   null

Command exit status:   127
</code></pre>
<p>I tried many things such as <code>${ output_file_name }</code>, <code>{ output_file_name }</code>, <code>$output_file_name</code> but in vain.</p>
<p>I'd like to improve few things in code:</p>
<p>a) The iteration/loop over file in <code>datasetFile</code>   I think something like <code>each</code> could help, but I need order to be maintained</p>
<p>b) Order in which loop of <code>datasetFile</code> is created. For instance, I'd like to keep lanes L003, L004, L005 in this order. Or, L001, L002, L003 order, I hope this is clear.</p>
<p>I cannot move ahead or able to sort this by myself.</p>
<p>I do not quite understand how and when to use <code>.map{}</code></p>
<p>Edited: after output added tuple for val and <code>path</code>. I still get same error.</p>

<nextflow>
nextflow: merge lane files from samples forward and reverse read
# nextflow: merge lane files from samples forward and reverse read

(Asked by: username_0 on 2023-07-19)

I'd like to merge files from different lanes for forward and reverse reads. For e.g., if a sample A has three lanes in forward, and three reverse, ultimately one reverse and one forward is to be generated.  I have normal and tumor for each sample. N as normal and T as tumor.
The lanes files are for e.g. as:
MM-265-DNA-N-01-01_L003_R1_001.fastq.gz
MM-265-DNA-N-01-01_L003_R2_001.fastq.gz
MM-265-DNA-N-01-01_L004_R1_001.fastq.gz
MM-265-DNA-N-01-01_L004_R2_001.fastq.gz

MM-265-DNA-T-01-01_L003_R1_001.fastq.gz
MM-265-DNA-T-01-01_L003_R2_001.fastq.gz
MM-265-DNA-T-01-01_L004_R1_001.fastq.gz
MM-265-DNA-T-01-01_L004_R2_001.fastq.gz

MM-465-DNA-N-01-01_L003_R1_001.fastq.gz
MM-465-DNA-N-01-01_L003_R2_001.fastq.gz
MM-465-DNA-N-01-01_L004_R1_001.fastq.gz
MM-465-DNA-N-01-01_L004_R2_001.fastq.gz

MM-465-DNA-T-01-01_L003_R1_001.fastq.gz
MM-465-DNA-T-01-01_L003_R2_001.fastq.gz
MM-465-DNA-T-01-01_L004_R1_001.fastq.gz
MM-465-DNA-T-01-01_L004_R2_001.fastq.gz  

The final output needs to be as:
MM-465_normal_R1.fastq MM-465_normal_R2.fastq
MM-465_tumor_R1.fastq MM-465_tumor_R2.fastq
MM-265_normal_R1.fastq MM-265_normal_R2.fastq
MM-265_tumor_R1.fastq MM-265_tumor_R2.fastq

I've code below broken in three parts:

regular expression
process that iterates over files
workflow that calls the above two code blocks

regex:
def get_sample_id( file ) {
def matcher = (file =~ /(MM-\d+)-DNA-N-.+/)
    if (matcher) {
        return matcher[0][1]
    } else {
        return ''
    }
}

process:
process file_print {

debug true

input:

tuple val(sample_id), path(datasetFile)
val(sample_status) 
val(read_value)

output:
publishDir '/arion/learn_new_concepts/', mode:'copy'

tuple val (sample_id), path("${output_file_name}")

script:

output_file_name = "${sample_id}_${sample_status}_${read_value}.fastq"

for (g in datasetFile){
println(g)

"zcat ${g} >> ${output_file_name} "

}

}

Workflow:
workflow {

normal_reads=Channel.fromFilePairs("/sc/arion/MM-*-N-*_L00*_R{1,2}_*.fastq.gz")
//MM-2658-DNA-N-01-01_L004_R2_001.fastq.gz  //

normal_forward_reads=normal_reads
        .map{ file -> return tuple( get_sample_id(file), file[1][0] )}.groupTuple(sort:true ) 

file_print(normal_forward_reads,"normal","R1").view() /// similar for reverse [1][1], R2 .. for tumor pass reads, "tumor", "R1"/R2"

}

I have not put code for reverse, but I get error as:
Caused by:   Process `file_print (1)` terminated with an error exit status (127) Command executed:   null

Command exit status:   127

I tried many things such as ${ output_file_name }, { output_file_name }, $output_file_name but in vain.
I'd like to improve few things in code:
a) The iteration/loop over file in datasetFile   I think something like each could help, but I need order to be maintained
b) Order in which loop of datasetFile is created. For instance, I'd like to keep lanes L003, L004, L005 in this order. Or, L001, L002, L003 order, I hope this is clear.
I cannot move ahead or able to sort this by myself.
I do not quite understand how and when to use .map{}
Edited: after output added tuple for val and path. I still get same error.

**Tags:** nextflow

---

## 1 answers

### Answer by username_1 on 2023-07-20. Score: 2 | Accepted answer

I think you are on the right track using groupTuple(sort: true). Another way to get what you want would be to instead parse the fromFilePairs grouping key. Unless you're looking to do more than just concatenate (and decompress) FASTQ files, I think all the process really requires is an output prefix (in addition to the FASTQ files of course), for example:
params.reads = "/sc/arion/MM-*_R{1,2}_001.fastq.gz"


process concat_reads {

    tag { prefix }

    input:
    tuple val(prefix), path(reads, stageAs: 'reads/*')

    output:
    tuple val(prefix), path("${prefix}.fastq")

    """
    zcat -f ${reads} > "${prefix}.fastq"
    """
}

workflow {

    Channel.fromFilePairs( params.reads ) \
        | branch { key, fastqs ->

            def matcher = key =~ /^(MM-\w+)-DNA-(\w+)-.*_L(\d+).*$/

            matches: matcher
                def (_, sample_id, sample_type, lane) = matcher[0].findAll()

                def new_sample_id = "${sample_id}_${sample_type}"

                return tuple( new_sample_id, lane as int, *fastqs )

            other: true
                return tuple( key, *fastqs )
        } \
        | set { samples }

    samples.matches \
        | groupTuple( sort: true ) \
        | flatMap { sample_id, lanes, fq1s, fq2s ->
            [
                tuple( "${sample_id}_R1", fq1s ),
                tuple( "${sample_id}_R2", fq2s ),
            ]
        } \
        | concat_reads \
        | branch { prefix, fastq ->

            def idx = prefix.lastIndexOf('_')

            def sample_id = prefix.substring(0, idx)
            def suffix = prefix.substring(idx + 1)

            read1: suffix == 'R1'
                return tuple( sample_id, fastq )
            read2: suffix == 'R2'
                return tuple( sample_id, fastq )
        } \
        | set { result }

    result.read1 \
        | join( result.read2 ) \
        | map { sample_id, fastq1, fastq2 ->

            tuple( sample_id, [ fastq1, fastq2 ] )
        } \
        | view()
}

Results:
$ nextflow run main.nf
N E X T F L O W  ~  version 23.04.1
Launching `main.nf` [jolly_noether] DSL2 - revision: 4a81f7a875
executor >  local (8)
[b8/a0b534] process > concat_reads (MM-265_T_R2) [100%] 8 of 8 ✔
[MM-465_T, [/path/to/work/9c/8b6fd328c848379423a0aba098cb74/MM-465_T_R1.fastq, /path/to/work/34/913780b1da902b5035f3bf56aac038/MM-465_T_R2.fastq]]
[MM-465_N, [/path/to/work/69/595120a1ee0a42217d06855ca9d1bc/MM-465_N_R1.fastq, /path/to/work/8f/31c406c5180073de577502fdb88fbf/MM-465_N_R2.fastq]]
[MM-265_N, [/path/to/work/1b/9c17cefb5aa51b3ee69832dc65d915/MM-265_N_R1.fastq, /path/to/work/d0/aac9f46e4a7f7165693b2537ad5630/MM-265_N_R2.fastq]]
[MM-265_T, [/path/to/work/d5/6295ff109df3ccb829a6f0a97b6e45/MM-265_T_R1.fastq, /path/to/work/b8/a0b53499ca13a523d811ecf772449b/MM-265_T_R2.fastq]]

Note that unless the FASTQ files are small, I would try to avoid decompressing the FASTQ files if possible. Most bioinformatics tools that deal with large FASTQ files will likely have gzip support, meaning you can replace the zcat -f with a cat in the example above (and remember to add the .gz extension to the output filename).

> **Comments:**
> **username_0**: Thank you for your reply and code. I'm sorry, I cannot understand what you did in the code as it is scary and complicated for me. May I know what is wrong in my code? I've .fastq extension as a suffix. May be a little explanation of the steps you did would be highly helpful to me. For example, when to `branch` branch.  Or, how do I put `normal` in the concatenated files?  
> **username_0**: what did you do after `concat_read  \ | branch` what is this doing? Why do we need this?  
> **username_0**: Hi I've edited my code for output by providing `tuple val` and `path` in parenthesis. However I get same error. The error messages don't convey any meaning, they are incredibly cryptic. :(  
> **username_1**: @DeathMetal I suspect your error message is being truncated because it tells us exactly why it failed: `.command.sh: line 2: null: command not found`. Try running your workflow instead using `nextflow run main.nf -ansi-log false`. Since your script block returns `null`, this becomes the command Nextflow tries to run for better or worse. I think you could replace this with a script block containing: `zcat -f ${datasetFile} > "${output_file_name}"` to get it running. Also, use `normal_reads.map{ key, files ->` to unpack the `fromFilePairs` output.  
> **username_1**: @DeathMetal Note that processes do not necessarily _process_ items in the order that they are received. For example, `concat_reads` can return results in any order. So, if we want to get back the reads where the R1 and R2 files are the first and second in the pair, respectively, then we need some way to separate the outputs so they can be joined correctly. In the above, I just used [`branch`](https://www.nextflow.io/docs/latest/operator.html#branch) to create two channels: `read1` and `read2` and then a [`join`](https://www.nextflow.io/docs/latest/operator.html#join) to put them back together.  
> **username_1**: @DeathMetal Also, if you wanted to replace `N` with `normal` and `T` with `tumor`, I think the place to do this would when defining the `new_sample_id` in the above. You could use a `Map` for example.  
> **username_0**: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/147354/discussion-between-death-metal-and-steve).  
> **username_0**: I am unable to understand `def (_, sample_id, sample_type, lane) = matcher[0].findAll()` why there is a `_` Is there a way to print/echo these outputs?  
> **username_0**: I want output concatenated from  `concat_reads` process only. They are generated in the folder as required. Is it necessary to write/do code below it where you `join`?  
> **username_1**: @DeathMetal If you just need the output of `concat_reads`, the code after and including the last `branch` statement is not necessary. But usually you would want to do something with these new pairs of files, which is why I included some code to show how this could be done. In reality, there are lots of ways to do this though. The underscore is just a placeholder for the full match. In Python and Groovy multi-assignment, at least, it is convention is to use an underscore for un-used variables. Use `samples.matches.view()` or insert a `println(your_variable)` into the `branch` statement to debug  

---

