<p>The <a href="https://www.nextflow.io/docs/latest/operator.html#merge" rel="nofollow noreferrer">merge operator</a> is deprecated and will be removed from future Nextflow releases. You'll need instead the <a href="https://www.nextflow.io/docs/latest/operator.html#join" rel="nofollow noreferrer">join operator</a> to join by chromosome. You can use the <a href="https://www.nextflow.io/docs/latest/channel.html#fromfilepairs" rel="nofollow noreferrer">fromFilePairs operator</a> to quickly get the group key, formatting it as required:</p>
<pre><code>channel
    .fromFilePairs(&quot;/some/path/HRC.r1-1.EGA.GRCh37.chr*.{hap.gz,legend.gz,samples}&quot;, size: 3)
    .map { group_key, file_list -&gt;
        tuple( group_key.replaceFirst(/^HRC\.r1-1\.EGA\.GRCh37\./, &quot;&quot;), file_list )
    }
    .set { ref_panel_files }


channel
    .fromFilePairs(&quot;/some/path/genetic_map_chr*_combined_b37.txt&quot;, size: 1)
    .map { group_key, file_list -&gt;
        tuple( group_key.replaceFirst(/^genetic_map_/, &quot;&quot;), file_list.first() )
    }
    .set { genetic_map_files }


qc1_chr
    .map { chrom, bfiles -&gt;
        tuple( chrom, bfiles.findAll { it.baseName.endsWith(&quot;step10&quot;) } )
    }
    .join( ref_panel_files )
    .join( genetic_map_files )
    .set { phasing_inputs }


process phasing {

    input:
    tuple val(chrom), path(input_bed), path(input_ref), path(input_map) from phasing_inputs

    script:
    def (bed, bim, fam) = input_bed
    def (haplotypes, legend, sample) = input_ref

    &quot;&quot;&quot;
    shapeit2 \\
        --thread &quot;<span class="math-container">${task.cpus}" \\
        --input-bed "$</span>{bed}&quot; &quot;<span class="math-container">${bim}" "$</span>{fam}&quot; \\
        --input-ref &quot;<span class="math-container">${haplotypes}" "$</span>{legend}&quot; &quot;<span class="math-container">${sample}" \\
        --input-map "$</span>{input_map}&quot; \\
        --duohmm \\
        --output-max &quot;${chrom}.phased&quot; \\
        --seed 54321
    &quot;&quot;&quot;
}
</code></pre>

CC BY-SA 4.0
2021-04-05T16:11:10.787Z
Thanks `@Steve`. Only one thing, the prefixes of `hap.gz, legend.gz and samples` are not same for each chromosome. That's why in my code there is a `*` at the beginning and another `*` at the chromosome label.
zillur rahman
CC BY-SA 4.0
2021-04-05T17:56:30.347Z
Hi `@Steve` I edited my question adding picture of running script and code snippet.
zillur rahman
CC BY-SA 4.0
2021-04-05T22:58:20.393Z
@zillurrahman: I think, though, you understand the general gist of the answer - that is, to create a tuple with a key as the first element which can then be used to join (multiple) channels. The above is just an example that uses the chromosome name as the join key. You should *not* expect a drop-in replacement for your code. I don't know what files your glob patterns match, and still don't. Your patterns could match multiple files, but I assumed you had one (hap, legend, sample) per chromosome. If the reference panel files have different prefixes, you may prefer separate channels for these.
Steve
CC BY-SA 4.0
2021-04-06T09:16:38.370Z
Sorry for the inconveniences. I rechecked the code, there were some type. Modified them and can see joined items. But getting a new error. `No such variable: chrom` But I declared the `chrom` variable in the input.
zillur rahman
CC BY-SA 4.0
2021-04-06T09:27:16.933Z
The new code snippet was added to my above question.
zillur rahman
CC BY-SA 4.0
2021-04-06T11:45:31.427Z
@zillurrahman I can't reproduce the error you get using your new code snippet. It should have failed earlier because of this typo: `path(intput_ref)` should be `path(input_ref)`. Do you know what line the error is being produced on?
Steve
CC BY-SA 4.0
2021-04-06T16:09:05.687Z
Thanks `@Steve`. The error is occurring at line 20 where the output is declared.
zillur rahman
CC BY-SA 4.0
2021-04-06T16:30:06.023Z
In the `.nextflow.log` file, it is showing that the error is occurring at line 20. And line 20 is `tuple val("${chrom}"),path("${chrom}.{phased.haps,phased.sample,log}") into phased` where the output was declared.
zillur rahman
CC BY-SA 4.0
2021-04-06T16:56:31.783Z
`@Steve` you are right. There was a type in my code. The error is `Unknown variable 'input_ref`. The typo is fixed and the code is running now! Thanks a lot!
zillur rahman
CC BY-SA 4.0
2021-04-06T18:41:57.247Z
If I run `phasing` separately, it works. But in combination with `prephasing`  it does not work. I mean bed|bim|fam files from channel from FilePairs works but from the output channel of previous process does not. Even I can not see the joined channel.view() in `.command.out` in the work directory. No error also. not sure what goes wrong. I added the updated code snippet in the above question.
zillur rahman
CC BY-SA 4.0
2021-04-07T01:33:07.713Z
@zillurrahman: I think your ouput declaration for qc1_chr needs to be: `tuple val("chr${chrom}"),path("chr${chrom}*.{bed,bim,fam}") into qc1_chr`. HTH.
Steve
CC BY-SA 4.0
2021-04-07T07:12:51.223Z
`@Steve` Thanks. I updated the output declaration of process `prephasing`. Still the process `phasing` does not work nor any error message. In the last screenshot of the above question, you can see that there is a work directory (at the left) assigned for `prephasing` process but nothing for `phasing` process. Not sure what I am doing wrong. The same `phasing` process works if I run it in a separate script where input `bed|bim|fam`  are assigned from `FilePairs`.
zillur rahman
CC BY-SA 4.0
2021-04-07T10:39:34.910Z
@zillurrahman: I suspect one of your input channels to join on isn't working as expected. Could you please post the output of `qc1_chr.view()`, `ref_panel.view()` and  `genetic_map.view()`? Thanks!
Steve
CC BY-SA 4.0
2021-04-07T17:27:06.610Z
Thanks `@Steve`. `ref_panel.view()` gives `[chr,[respective hap.gz,legend.gz,sample.gz]]`, `genetic_map.view()` gives `[chr,[respective maps]]`. The problem with filtering `qc1_chr`. `qc1_chr.filter{chrom, bfiles -> bfiles.collect {it.getBaseName()}.every {it.endsWith("step10")}}.view()`  shows nothing. Interestingly, same filtering method works when the `bed|bim|fam` files were from `FilePairs`. I also tried `qc1_chr.filter { it[1].name.contains('step10') }.view()` but nothing. `qc1_chr.view()` gives `[chr1-22,[respective all plink files]]`
zillur rahman
CC BY-SA 4.0
2021-04-07T23:17:19.260Z
My mistake, I think you'll need to replace the filter statement with `.map { chrom, bfiles -> tuple( chrom, bfiles.findAll { it.baseName.endsWith("step10") } ) }`. No filtering is required, we just need to select out the required step10 files.
Steve
CC BY-SA 4.0
2021-04-08T06:22:05.767Z
Thank you very much `@Steve`. It works now. as there are multiple patterns, I used `channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/HRC/haplegendsample/*hrc.r1-1.ega.grch37.chr*.{hap.gz,legend.gz,samples}',size:3).map {group_key,file_list -> tuple(gro
up_key.replaceAll(/^.*.hrc\.r1-1\.ega\.grch37\.|\.haplotypes\.noibd|\.haplotypes/,""),file_list)}.set {ref_panel}`. Do you think `replaceAll` is ok instead of `replaceFirst` . I got expected output using using `replaceAll`. Could I use multiple pattern using `replaceFirst`?
zillur rahman
CC BY-SA 4.0
2021-04-08T10:44:21.393Z
@zillurrahman: I don't think you can since the first looks like a prefix and the last two look like suffixes. I think you'd need two replaceFirst calls. I'd be tempted to just use the first string that looks like a chromosome name: `group_key.find(~/chr\d{1,2}/)`. HTH.
Steve
CC BY-SA 4.0
2021-04-08T11:20:26.200Z
Thanks `@Steve` I just tested the `find` option and it works. Thanks a lot! It seems fewer code and easier to write!
zillur rahman
CC BY-SA 4.0
2021-04-10T07:57:30.830Z
Hi `@Steve`, Wondering if you get a chance to look at https://bioinformatics.stackexchange.com/questions/15698/parse-csv-text-file-to-create-arrays-in-nextflow . I just need a little help to create an array from a csv file in nextflow.
zillur rahman
CC BY-SA 4.0
2021-04-05T14:40:11.413Z
2021-04-07T23:18:54.153Z
2021-04-07T23:18:54.153Z
Steve
<p>I need to merge multiple channels each emits multiple file. So far I did something as below:</p>
<pre><code>process prephasing {
publishDir params.out, mode:'copy'
input:
tuple val(prefix), path(bfiles) from plink_data
each chrom from chroms
path hrc_sites_file
output:
tuple val(&quot;<span class="math-container">${chrom}"),path("chr$</span>{chrom}*.{bed,bim,fam}&quot;) into qc1_chr
&quot;&quot;&quot;
PRE_PASHING_QC2.sh &quot;<span class="math-container">${hrc_sites_file}" "$</span>{prefix}&quot; &quot;${chrom}&quot;
&quot;&quot;&quot;
}
channel.fromPath('/some/path/genetic_map_chr*_combined_b37.txt').into {shape_map;i
mp_map}
channel.fromPath('/some/path/*hrc.r1-1.ega.grch*.hap.gz').into {shape_ref;
imp_ref}
channel.fromPath('/some/path/*hrc.r1-1.ega.grch37.chr*.legend.gz').into {s
hape_legend;imp_legend}
channel.fromPath('/some/path/*hrc.r1-1.ega.grch37.chr*.samples').into {sha
pe_sample;imp_sample}
qc1_chr.filter{name, bfiles -&gt; bfiles.collect {it.getBaseName()}.every {it.endsWith(&quot;step10&quot;)}}.into {shape_in;imp_in}

shape_para=shape_in.merge(shape_map).merge(shape_ref).merge(shape_legend).merge(shape_sample)
    
process phasing {
publishDir params.out, mode:'copy'
input:
each file(penta) from shape_para
output:
path(&quot;*step10*&quot;) into phased
script:
def (in, map, ref, legend, sample) = penta
&quot;&quot;&quot;
shapeit2 -T 32 --input-bed &quot;<span class="math-container">${in}" --input-map "$</span>{map}&quot; --input-ref &quot;<span class="math-container">${ref}" "$</span>{legend}&quot; &quot;<span class="math-container">${sample}" --duohmm -O "$</span>{in}.phased&quot; --seed 54321
&quot;&quot;&quot;
}
</code></pre>
<p>The &quot;prephasing&quot; process works fine and produce expected output. But the &quot;phasing&quot; process does not produce any output nor any error. Any help?</p>
<p>I have tried according to <code>@Steve</code> as below:</p>
<pre><code>channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/1000Genomes/genetic_map_chr*_combined_b37.txt',size:1).map {group_key, file_list -&gt; tuple(group_key.replaceFirst(/^genetic_map_/,&quot;&quot;),file_list.first())}.set {genetic_map}

channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/HRC/haplegendsample/*hrc.r1-1.ega.grch37.chr*.haplotypes.{hap.gz,legend.gz,samples}',size:3).map{group_key,file_list -&gt; tuple(group_key.replaceFirst(/^*HRC\.r1-1\.ega\.grch37\./,&quot;&quot;),file_list)}.set {ref_panel}

qc1_chr.filter{chrom, bfiles -&gt; bfiles.collect {it.getBaseName()}.every {it.endsWith(&quot;step10&quot;)}}.join(ref_panel).join(genetic_map).into {shape_in;imp_in}

process phasing {
publishDir params.out, mode:'copy'
input:
tuple val(chrom), path(input_bed), path(intput_ref), path(input_map) from shape_in
output:
tuple val(&quot;<span class="math-container">${chrom}"),path("$</span>{chrom}.{phased.haps,phased.sample,log}&quot;) into phased
script:
def (bed, bim, fam) = input_bed
def (haplotype, legend, sample) = input_ref
&quot;&quot;&quot;
shapeit2 --thread &quot;<span class="math-container">${task.cpus}" --input-bed "$</span>{bed}&quot; &quot;<span class="math-container">${bim}" "$</span>{fam}&quot; --input-map &quot;<span class="math-container">${input_map}" --input-ref "$</span>{haplotype}&quot; &quot;<span class="math-container">${legend}" "$</span>{sample}&quot; --duohmm --output-max &quot;${chrom}.phased&quot; --seed 54321
&quot;&quot;&quot;
}
</code></pre>
<p>But getting no output so far, (attached image<a href="https://i.sstatic.net/B1PtL.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/B1PtL.png" alt="phasing process" /></a>)</p>
<p>I have tried to check individual channel separately using channel.view(), identified the problems, modified the code and checked again. Using the code below I can see individual channel emits 22 lines [chr1-22,[file_list]]. After merging I can see [chr1-22,[joined_file_list].</p>
<pre><code>channel
    .fromFilePairs( &quot;output3/*.{bed,bim,fam}&quot;, size:3 )
    .set { plink_data }
plink_data.filter{chrom, bfiles -&gt; bfiles.collect {it.getBaseName()}.every {it.endsWith(&quot;step10&quot;)}}.map{group_key,file_list -&gt; tuple (group_key.
replaceFirst(/^*\.step10/,&quot;&quot;), file_list)}.set {in_pl}

channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/1000Genomes/genetic_map_chr*_combined_b37.txt',size:1).map {
group_key, file_list -&gt; tuple(group_key.replaceFirst(/^*genetic_map_/,&quot;&quot;),file_list.first())}.set {genetic_map}
//genetic_map.view()

channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/HRC/haplegendsample/*hrc.r1-1.ega.grch37.chr*.haplotypes.{ha
p.gz,legend.gz,samples}',size:3).map {group_key,file_list -&gt; tuple(group_key.replaceFirst(/^.*.hrc\.r1-1\.ega\.grch37\./,&quot;&quot;),file_list)}.set {re
f_panel}
//ref_panel.view()



in_pl.join(ref_panel).join(genetic_map).into {shape_in;imp_in}
process phasing {
//publishDir params.out, mode:'copy'
input:
tuple val(chrom), path(input_bed), path(intput_ref), path(input_map) from shape_in
output:
tuple val(&quot;<span class="math-container">${chrom}"),path("$</span>{chrom}.{phased.haps,phased.sample,log}&quot;) in
to phased
script:
def (bed, bim, fam) = input_bed
def (haplotype, legend, sample) = input_ref
&quot;&quot;&quot;
//shapeit=&quot;/cluster/projects/p697/projects/moba_qc_imputation/software/shap
eit&quot;
shapeit --thread &quot;<span class="math-container">${task.cpus}" --input-bed "$</span>{bed}&quot; &quot;<span class="math-container">${bim}" "$</span>{fam}&quot; --input-map &quot;<span class="math-container">${input_map}" --input-ref "$</span>{haplotype}&quot; &quot;<span class="math-container">${legend}" "$</span>{samp
le}&quot; --duohmm --output-max &quot;${chrom}.phased&quot; --seed 54321
&quot;&quot;&quot;
}
</code></pre>
<p><a href="https://i.sstatic.net/CR2sW.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/CR2sW.jpg" alt="joined channel example view" /></a>
But now I get error:</p>
<pre><code>No such variable: chrom
</code></pre>
<p>But I decalred &quot;chrom&quot; in the input</p>
<p>The input bed|bim|fam files from channel from FilePairs works but from a output channel of previous process does not. If I run <code>phasing</code> solo and all the inputs from channel from FilePairs it works. But if I run <code>prephasing</code> and <code>phasing</code> together, the <code>prephasing</code> process works but for <code>phasing</code> process not output and no error; Not sure what I am doing wrong.</p>
<pre><code>hrc_sites_file = file(params.hrc_sites_file)

Channel.fromFilePairs( &quot;${params.input_dir}/*.{bed,bim,fam}&quot;, size:3 ).set { plink_data }

Channel.of(1..22).set { chroms }
params.out='output/'

process prephasing {
publishDir params.out, mode:'copy'
input:
tuple val(prefix), path(bfiles) from plink_data
each chrom from chroms
path hrc_sites_file
output:
tuple val(&quot;<span class="math-container">${chrom}"),path("chr$</span>{chrom}*.{bed,bim,fam}&quot;) into qc1_chr
tuple val(&quot;<span class="math-container">${chrom}"),path("chr$</span>{chrom}*log&quot;) into qc1_log
tuple val(&quot;<span class="math-container">${chrom}"),path("*-chr$</span>{chrom}.step4-HRC.txt&quot;) into qc1_hrc
path(&quot;chr<span class="math-container">${chrom}.step1b.snplist.txt") into qc1_snps
path("chr$</span>{chrom}.step1a.dupvar&quot;) into qc1_dups
&quot;&quot;&quot;
PRE_PASHING_QC2.sh &quot;<span class="math-container">${hrc_sites_file}" "$</span>{prefix}&quot; &quot;${chrom}&quot;
&quot;&quot;&quot;
}
channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/1000Genomes/genetic_map_chr*_combined_b37.txt',size:1).map {group_key, file_list -&gt; tuple(group_key.replaceFirst(/^*gen
etic_map_/,&quot;&quot;),file_list.first())}.set {genetic_map}

channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/HRC/haplegendsample/*hrc.r1-1.ega.grch37.chr*.haplotypes.{hap.gz,legend.gz,samples}',size:3).map{group_key,file_list -&gt;
 tuple(group_key.replaceFirst(/^.*.hrc\.r1-1\.ega\.grch37\./,&quot;&quot;),file_list)}.set {ref_panel}

qc1_chr.filter{chrom, bfiles -&gt; bfiles.collect {it.getBaseName()}.every {it.endsWith(&quot;step10&quot;)}}.map{group_key,file_list -&gt; tuple (group_key.replaceFirst(/^*\.step10/,&quot;&quot;), file_list)}.join(ref_panel).joi
n(genetic_map).into {shape_in;imp_in}
imp_in.view()
process phasing {
publishDir params.out, mode:'copy'
input:
tuple val(chrom), path(input_bed), path(input_ref), path(input_map) from shape_in
output:
tuple val(&quot;<span class="math-container">${chrom}"),path("$</span>{chrom}.*&quot;) into phased
script:
def (bed, bim, fam) = input_bed
def (haplotype, legend, sample) = input_ref
&quot;&quot;&quot;
shapeit --thread &quot;<span class="math-container">${task.cpus}" --input-bed "$</span>{bed}&quot; &quot;<span class="math-container">${bim}" "$</span>{fam}&quot; --input-map &quot;<span class="math-container">${input_map}" --input-ref "$</span>{haplotype}&quot; &quot;<span class="math-container">${legend}" "$</span>{sample}&quot; --duohmm --output-max &quot;${chrom}.step10&quot; --seed 54321
&quot;&quot;&quot;
}
</code></pre>
<p><a href="https://i.sstatic.net/bjxQJ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/bjxQJ.png" alt="prephasing works but phasing does not" /></a></p>

<nextflow>
Merge multiple channels each emits multiple files in nextflow
# Merge multiple channels each emits multiple files in nextflow

(Asked by: username_0 on 2021-04-05)

I need to merge multiple channels each emits multiple file. So far I did something as below:
process prephasing {
publishDir params.out, mode:'copy'
input:
tuple val(prefix), path(bfiles) from plink_data
each chrom from chroms
path hrc_sites_file
output:
tuple val("${chrom}"),path("chr${chrom}*.{bed,bim,fam}") into qc1_chr
"""
PRE_PASHING_QC2.sh "${hrc_sites_file}" "${prefix}" "${chrom}"
"""
}
channel.fromPath('/some/path/genetic_map_chr*_combined_b37.txt').into {shape_map;i
mp_map}
channel.fromPath('/some/path/*hrc.r1-1.ega.grch*.hap.gz').into {shape_ref;
imp_ref}
channel.fromPath('/some/path/*hrc.r1-1.ega.grch37.chr*.legend.gz').into {s
hape_legend;imp_legend}
channel.fromPath('/some/path/*hrc.r1-1.ega.grch37.chr*.samples').into {sha
pe_sample;imp_sample}
qc1_chr.filter{name, bfiles -> bfiles.collect {it.getBaseName()}.every {it.endsWith("step10")}}.into {shape_in;imp_in}

shape_para=shape_in.merge(shape_map).merge(shape_ref).merge(shape_legend).merge(shape_sample)
    
process phasing {
publishDir params.out, mode:'copy'
input:
each file(penta) from shape_para
output:
path("*step10*") into phased
script:
def (in, map, ref, legend, sample) = penta
"""
shapeit2 -T 32 --input-bed "${in}" --input-map "${map}" --input-ref "${ref}" "${legend}" "${sample}" --duohmm -O "${in}.phased" --seed 54321
"""
}

The "prephasing" process works fine and produce expected output. But the "phasing" process does not produce any output nor any error. Any help?
I have tried according to @Steve as below:
channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/1000Genomes/genetic_map_chr*_combined_b37.txt',size:1).map {group_key, file_list -> tuple(group_key.replaceFirst(/^genetic_map_/,""),file_list.first())}.set {genetic_map}

channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/HRC/haplegendsample/*hrc.r1-1.ega.grch37.chr*.haplotypes.{hap.gz,legend.gz,samples}',size:3).map{group_key,file_list -> tuple(group_key.replaceFirst(/^*HRC\.r1-1\.ega\.grch37\./,""),file_list)}.set {ref_panel}

qc1_chr.filter{chrom, bfiles -> bfiles.collect {it.getBaseName()}.every {it.endsWith("step10")}}.join(ref_panel).join(genetic_map).into {shape_in;imp_in}

process phasing {
publishDir params.out, mode:'copy'
input:
tuple val(chrom), path(input_bed), path(intput_ref), path(input_map) from shape_in
output:
tuple val("${chrom}"),path("${chrom}.{phased.haps,phased.sample,log}") into phased
script:
def (bed, bim, fam) = input_bed
def (haplotype, legend, sample) = input_ref
"""
shapeit2 --thread "${task.cpus}" --input-bed "${bed}" "${bim}" "${fam}" --input-map "${input_map}" --input-ref "${haplotype}" "${legend}" "${sample}" --duohmm --output-max "${chrom}.phased" --seed 54321
"""
}

But getting no output so far, (attached image)
I have tried to check individual channel separately using channel.view(), identified the problems, modified the code and checked again. Using the code below I can see individual channel emits 22 lines [chr1-22,[file_list]]. After merging I can see [chr1-22,[joined_file_list].
channel
    .fromFilePairs( "output3/*.{bed,bim,fam}", size:3 )
    .set { plink_data }
plink_data.filter{chrom, bfiles -> bfiles.collect {it.getBaseName()}.every {it.endsWith("step10")}}.map{group_key,file_list -> tuple (group_key.
replaceFirst(/^*\.step10/,""), file_list)}.set {in_pl}

channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/1000Genomes/genetic_map_chr*_combined_b37.txt',size:1).map {
group_key, file_list -> tuple(group_key.replaceFirst(/^*genetic_map_/,""),file_list.first())}.set {genetic_map}
//genetic_map.view()

channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/HRC/haplegendsample/*hrc.r1-1.ega.grch37.chr*.haplotypes.{ha
p.gz,legend.gz,samples}',size:3).map {group_key,file_list -> tuple(group_key.replaceFirst(/^.*.hrc\.r1-1\.ega\.grch37\./,""),file_list)}.set {re
f_panel}
//ref_panel.view()



in_pl.join(ref_panel).join(genetic_map).into {shape_in;imp_in}
process phasing {
//publishDir params.out, mode:'copy'
input:
tuple val(chrom), path(input_bed), path(intput_ref), path(input_map) from shape_in
output:
tuple val("${chrom}"),path("${chrom}.{phased.haps,phased.sample,log}") in
to phased
script:
def (bed, bim, fam) = input_bed
def (haplotype, legend, sample) = input_ref
"""
//shapeit="/cluster/projects/p697/projects/moba_qc_imputation/software/shap
eit"
shapeit --thread "${task.cpus}" --input-bed "${bed}" "${bim}" "${fam}" --input-map "${input_map}" --input-ref "${haplotype}" "${legend}" "${samp
le}" --duohmm --output-max "${chrom}.phased" --seed 54321
"""
}


But now I get error:
No such variable: chrom

But I decalred "chrom" in the input
The input bed|bim|fam files from channel from FilePairs works but from a output channel of previous process does not. If I run phasing solo and all the inputs from channel from FilePairs it works. But if I run prephasing and phasing together, the prephasing process works but for phasing process not output and no error; Not sure what I am doing wrong.
hrc_sites_file = file(params.hrc_sites_file)

Channel.fromFilePairs( "${params.input_dir}/*.{bed,bim,fam}", size:3 ).set { plink_data }

Channel.of(1..22).set { chroms }
params.out='output/'

process prephasing {
publishDir params.out, mode:'copy'
input:
tuple val(prefix), path(bfiles) from plink_data
each chrom from chroms
path hrc_sites_file
output:
tuple val("${chrom}"),path("chr${chrom}*.{bed,bim,fam}") into qc1_chr
tuple val("${chrom}"),path("chr${chrom}*log") into qc1_log
tuple val("${chrom}"),path("*-chr${chrom}.step4-HRC.txt") into qc1_hrc
path("chr${chrom}.step1b.snplist.txt") into qc1_snps
path("chr${chrom}.step1a.dupvar") into qc1_dups
"""
PRE_PASHING_QC2.sh "${hrc_sites_file}" "${prefix}" "${chrom}"
"""
}
channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/1000Genomes/genetic_map_chr*_combined_b37.txt',size:1).map {group_key, file_list -> tuple(group_key.replaceFirst(/^*gen
etic_map_/,""),file_list.first())}.set {genetic_map}

channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/HRC/haplegendsample/*hrc.r1-1.ega.grch37.chr*.haplotypes.{hap.gz,legend.gz,samples}',size:3).map{group_key,file_list ->
 tuple(group_key.replaceFirst(/^.*.hrc\.r1-1\.ega\.grch37\./,""),file_list)}.set {ref_panel}

qc1_chr.filter{chrom, bfiles -> bfiles.collect {it.getBaseName()}.every {it.endsWith("step10")}}.map{group_key,file_list -> tuple (group_key.replaceFirst(/^*\.step10/,""), file_list)}.join(ref_panel).joi
n(genetic_map).into {shape_in;imp_in}
imp_in.view()
process phasing {
publishDir params.out, mode:'copy'
input:
tuple val(chrom), path(input_bed), path(input_ref), path(input_map) from shape_in
output:
tuple val("${chrom}"),path("${chrom}.*") into phased
script:
def (bed, bim, fam) = input_bed
def (haplotype, legend, sample) = input_ref
"""
shapeit --thread "${task.cpus}" --input-bed "${bed}" "${bim}" "${fam}" --input-map "${input_map}" --input-ref "${haplotype}" "${legend}" "${sample}" --duohmm --output-max "${chrom}.step10" --seed 54321
"""
}

**Tags:** nextflow

---

## 1 answers

### Answer by username_1 on 2021-04-05. Score: 3 | Accepted answer

The merge operator is deprecated and will be removed from future Nextflow releases. You'll need instead the join operator to join by chromosome. You can use the fromFilePairs operator to quickly get the group key, formatting it as required:
channel
    .fromFilePairs("/some/path/HRC.r1-1.EGA.GRCh37.chr*.{hap.gz,legend.gz,samples}", size: 3)
    .map { group_key, file_list ->
        tuple( group_key.replaceFirst(/^HRC\.r1-1\.EGA\.GRCh37\./, ""), file_list )
    }
    .set { ref_panel_files }


channel
    .fromFilePairs("/some/path/genetic_map_chr*_combined_b37.txt", size: 1)
    .map { group_key, file_list ->
        tuple( group_key.replaceFirst(/^genetic_map_/, ""), file_list.first() )
    }
    .set { genetic_map_files }


qc1_chr
    .map { chrom, bfiles ->
        tuple( chrom, bfiles.findAll { it.baseName.endsWith("step10") } )
    }
    .join( ref_panel_files )
    .join( genetic_map_files )
    .set { phasing_inputs }


process phasing {

    input:
    tuple val(chrom), path(input_bed), path(input_ref), path(input_map) from phasing_inputs

    script:
    def (bed, bim, fam) = input_bed
    def (haplotypes, legend, sample) = input_ref

    """
    shapeit2 \\
        --thread "${task.cpus}" \\
        --input-bed "${bed}" "${bim}" "${fam}" \\
        --input-ref "${haplotypes}" "${legend}" "${sample}" \\
        --input-map "${input_map}" \\
        --duohmm \\
        --output-max "${chrom}.phased" \\
        --seed 54321
    """
}

> **Comments:**
> **username_0**: Thanks `@Steve`. Only one thing, the prefixes of `hap.gz, legend.gz and samples` are not same for each chromosome. That's why in my code there is a `*` at the beginning and another `*` at the chromosome label.  
> **username_0**: Hi `@Steve` I edited my question adding picture of running script and code snippet.  
> **username_1**: @zillurrahman: I think, though, you understand the general gist of the answer - that is, to create a tuple with a key as the first element which can then be used to join (multiple) channels. The above is just an example that uses the chromosome name as the join key. You should *not* expect a drop-in replacement for your code. I don't know what files your glob patterns match, and still don't. Your patterns could match multiple files, but I assumed you had one (hap, legend, sample) per chromosome. If the reference panel files have different prefixes, you may prefer separate channels for these.  
> **username_0**: Sorry for the inconveniences. I rechecked the code, there were some type. Modified them and can see joined items. But getting a new error. `No such variable: chrom` But I declared the `chrom` variable in the input.  
> **username_0**: The new code snippet was added to my above question.  
> **username_1**: @zillurrahman I can't reproduce the error you get using your new code snippet. It should have failed earlier because of this typo: `path(intput_ref)` should be `path(input_ref)`. Do you know what line the error is being produced on?  
> **username_0**: Thanks `@Steve`. The error is occurring at line 20 where the output is declared.  
> **username_0**: In the `.nextflow.log` file, it is showing that the error is occurring at line 20. And line 20 is `tuple val("${chrom}"),path("${chrom}.{phased.haps,phased.sample,log}") into phased` where the output was declared.  
> **username_0**: `@Steve` you are right. There was a type in my code. The error is `Unknown variable 'input_ref`. The typo is fixed and the code is running now! Thanks a lot!  
> **username_0**: If I run `phasing` separately, it works. But in combination with `prephasing`  it does not work. I mean bed|bim|fam files from channel from FilePairs works but from the output channel of previous process does not. Even I can not see the joined channel.view() in `.command.out` in the work directory. No error also. not sure what goes wrong. I added the updated code snippet in the above question.  
> **username_1**: @zillurrahman: I think your ouput declaration for qc1_chr needs to be: `tuple val("chr${chrom}"),path("chr${chrom}*.{bed,bim,fam}") into qc1_chr`. HTH.  
> **username_0**: `@Steve` Thanks. I updated the output declaration of process `prephasing`. Still the process `phasing` does not work nor any error message. In the last screenshot of the above question, you can see that there is a work directory (at the left) assigned for `prephasing` process but nothing for `phasing` process. Not sure what I am doing wrong. The same `phasing` process works if I run it in a separate script where input `bed|bim|fam`  are assigned from `FilePairs`.  
> **username_1**: @zillurrahman: I suspect one of your input channels to join on isn't working as expected. Could you please post the output of `qc1_chr.view()`, `ref_panel.view()` and  `genetic_map.view()`? Thanks!  
> **username_0**: Thanks `@Steve`. `ref_panel.view()` gives `[chr,[respective hap.gz,legend.gz,sample.gz]]`, `genetic_map.view()` gives `[chr,[respective maps]]`. The problem with filtering `qc1_chr`. `qc1_chr.filter{chrom, bfiles -> bfiles.collect {it.getBaseName()}.every {it.endsWith("step10")}}.view()`  shows nothing. Interestingly, same filtering method works when the `bed|bim|fam` files were from `FilePairs`. I also tried `qc1_chr.filter { it[1].name.contains('step10') }.view()` but nothing. `qc1_chr.view()` gives `[chr1-22,[respective all plink files]]`  
> **username_1**: My mistake, I think you'll need to replace the filter statement with `.map { chrom, bfiles -> tuple( chrom, bfiles.findAll { it.baseName.endsWith("step10") } ) }`. No filtering is required, we just need to select out the required step10 files.  
> **username_0**: Thank you very much `@Steve`. It works now. as there are multiple patterns, I used `channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/HRC/haplegendsample/*hrc.r1-1.ega.grch37.chr*.{hap.gz,legend.gz,samples}',size:3).map {group_key,file_list -> tuple(gro up_key.replaceAll(/^.*.hrc\.r1-1\.ega\.grch37\.|\.haplotypes\.noibd|\.haplotypes/,""),file_list)}.set {ref_panel}`. Do you think `replaceAll` is ok instead of `replaceFirst` . I got expected output using using `replaceAll`. Could I use multiple pattern using `replaceFirst`?  
> **username_1**: @zillurrahman: I don't think you can since the first looks like a prefix and the last two look like suffixes. I think you'd need two replaceFirst calls. I'd be tempted to just use the first string that looks like a chromosome name: `group_key.find(~/chr\d{1,2}/)`. HTH.  
> **username_0**: Thanks `@Steve` I just tested the `find` option and it works. Thanks a lot! It seems fewer code and easier to write!  
> **username_0**: Hi `@Steve`, Wondering if you get a chance to look at https://bioinformatics.stackexchange.com/questions/15698/parse-csv-text-file-to-create-arrays-in-nextflow . I just need a little help to create an array from a csv file in nextflow.  

---

