<p>I suspect your <code>HRC_1000G_check_bim</code> and <code>hrc_sites_file</code> channels are <a href="https://www.nextflow.io/docs/latest/channel.html#queue-channel" rel="nofollow noreferrer">queue channels</a> when they should be <a href="https://www.nextflow.io/docs/latest/channel.html#value-channel" rel="nofollow noreferrer">value channels</a>. Please again see <a href="https://www.nextflow.io/docs/latest/process.html#understand-how-multiple-input-channels-work" rel="nofollow noreferrer">how multiple channels work</a>. Also, <a href="https://www.nextflow.io/docs/latest/sharing.html#third-party-scripts" rel="nofollow noreferrer">third-party scripts</a>, like your Perl script, would be better placed in a directory called <code>bin/</code> at the root of your project repository. Nextflow adds this directory to your $PATH, so scripts in this directory will be automatically accessible in your pipeline without needing to localize them using a separate channel. This way also avoids having to specify an absolute path to invoke them. Just make sure your script is executable: <code>chmod +x your_script.pl</code></p>
<pre><code>hrc_sites = file( params.hrc_sites_file )

process step5 {

    input:
    tuple val(prefix), val(chr_name), path(bfiles) from step4_frq
    path hrc_sites

    &quot;&quot;&quot;
    your_script.pl \\
        -b &quot;<span class="math-container">${prefix}$</span>{chr_name}.step4.bim&quot; \\
        -f &quot;<span class="math-container">${prefix}$</span>{chr_name}.step4.frq&quot; \\
        -r &quot;${hrc_sites}&quot; \\
        -h
    &quot;&quot;&quot;
}
</code></pre>

CC BY-SA 4.0
2021-03-25T19:02:21.590Z
Thank you `@Steve`. I just avoided using channel for those two files. I need to use them from absolute path, because other users also use this pipeline. However, it works now. One thing, I am using slurm executor with a config file, so far no problem with this. But I want to run the script in background (similar to sbatch submit, after submitting the job I will logout and the job will continue to run). How could I do this?
zillur rahman
CC BY-SA 4.0
2021-03-25T20:59:33.510Z
One thing, After finishing the jobs, transferring files to publish directory taking too much time. There are only 264 files but it is taking hours to transfer. Any solution for this?
zillur rahman
CC BY-SA 4.0
2021-03-25T23:55:01.860Z
No worries at all @zillurrahman. It's generally better to avoid specifying absolute paths in your process scripts. Avoiding absolute paths will mean that your process tasks can be run in isolation from each other and that your users could run your workflow on platforms like AWS Batch where there is no shared filesystem.
Steve
CC BY-SA 4.0
2021-03-26T00:04:13.097Z
I don't have experience with Slurm, but you should be able to submit a job non-interactively using [sbatch](https://support.ceci-hpc.be/doc/_contents/QuickStart/SubmittingJobs/SlurmTutorial.html#creating-a-job) like you said. If it's submitted non-interactively, you can logout and the job running your workflow will continue on.
Steve
CC BY-SA 4.0
2021-03-26T00:24:15.713Z
I'm not too sure about your last question; 264 files doesn't sound like a lot, but how much data are we talking? Could you be maxing out your network bandwidth? Are you writing to archival storage? Your sysadmin would be the right person to answer these sorts of questions and they might have some useful suggestions on how best run the workflow. For example, it might be better to write to a high performance disk array and then have a separate, but dependant, job copy these files to their final location.
Steve
CC BY-SA 4.0
2021-03-26T00:28:07.723Z
You may also want to consider other publishing modes and parameterizing these, for example: `mode: params.publish_mode`. Consider also parameterizing the 'enabled' option so to let your users decide what outputs they want: `enabled: params.publish_everything || params.publish_step5`. HTH.
Steve
CC BY-SA 4.0
2021-03-25T14:08:11.843Z
2021-03-25T14:08:11.843Z
Steve
<p>I was trying to develop a pipeline using nextflow and plink. My code runs without any error, but got only two output files instead of 22 (one for each chromosome). Here is part my code block, any help?</p>
<pre><code>process step5 {
publishDir params.publish, mode: 'copy'
input:
val(hrc) from HRC_1000G_check_bim
val(hrc_site) from hrc_sites_file
tuple val(prefix), val(chr_name), path(bfiles) from step4_frq
output:
tuple val(&quot;<span class="math-container">${prefix}"), val(chr_name), path("LOG-$</span>{prefix}<span class="math-container">${chr_name}.step4-HRC.txt") into step4_log
tuple val("$</span>{prefix}&quot;), val(chr_name), path(&quot;Exclude-<span class="math-container">${prefix}$</span>{chr_name}.step4-HRC.txt&quot;) into step4_exclude
tuple val(&quot;<span class="math-container">${prefix}"), val(chr_name), path("Chromosome-$</span>{prefix}<span class="math-container">${chr_name}.step4-HRC.txt") into step4_chromosome
tuple val("$</span>{prefix}&quot;), val(chr_name), path(&quot;Position-<span class="math-container">${prefix}$</span>{chr_name}.step4-HRC.txt&quot;) into step4_position
tuple val(&quot;<span class="math-container">${prefix}"), val(chr_name), path("Strand-Flip-$</span>{prefix}<span class="math-container">${chr_name}.step4-HRC.txt") into step4_strand
tuple val("$</span>{prefix}&quot;), val(chr_name), path(&quot;Force-Allele1-<span class="math-container">${prefix}$</span>{chr_name}.step4-HRC.txt&quot;) into step4_allele
tuple val(&quot;<span class="math-container">${prefix}"), val(chr_name), path("ID-$</span>{prefix}<span class="math-container">${chr_name}.step4-HRC.txt") into step4_id
script:
"""
perl "$</span>{hrc}&quot; -b &quot;<span class="math-container">${prefix}$</span>{chr_name}.step4.bim&quot; -f &quot;<span class="math-container">${prefix}$</span>{chr_name}.step4.frq&quot; -r &quot;${hrc_site}&quot; -h
&quot;&quot;&quot;
}
</code></pre>
<p>The output files are:</p>
<pre><code>gsa-out/Chromosome-chr22.step4-HRC.txt     gsa-out/Force-Allele1-chr3.step4-HRC.txt  gsa-out/Position-chr22.step4-HRC.txt
gsa-out/Chromosome-chr3.step4-HRC.txt      gsa-out/ID-chr22.step4-HRC.txt            gsa-out/Position-chr3.step4-HRC.txt
gsa-out/Exclude-chr22.step4-HRC.txt        gsa-out/ID-chr3.step4-HRC.txt             gsa-out/Strand-Flip-chr22.step4-HRC.txt
gsa-out/Exclude-chr3.step4-HRC.txt         gsa-out/LOG-chr22.step4-HRC.txt           gsa-out/Strand-Flip-chr3.step4-HRC.txt
gsa-out/Force-Allele1-chr22.step4-HRC.txt  gsa-out/LOG-chr3.step4-HRC.txt
</code></pre>
<p>I am not sure why I am getting only 2, instead of 22 output files. With similar output channel previously generated 22 output files in the same script. For eaxmple:</p>
<pre><code>process step1b {
publishDir params.publish, mode: 'copy'
input:
tuple val(prefix), val(chr_name), path(bfiles) from step1a_out
output:
tuple val(&quot;<span class="math-container">${prefix}"), val(chr_name), path("$</span>{prefix}<span class="math-container">${chr_name}.step1c.{bed,bim,fam}") into step1b_results
path("$</span>{prefix}<span class="math-container">${chr_name}.step1c.log") into step1b_logs
path("$</span>{prefix}${chr_name}.step1b.snplist.txt&quot;) into step1b_snplist
script:

&quot;&quot;&quot;
awk '{ if ((\\\$5==&quot;T&quot; &amp;&amp; \\\$6==&quot;A&quot;)||(\\\$5==&quot;A&quot; &amp;&amp; \\\$6==&quot;T&quot;)||(\\\$5==&quot;C&quot; &amp;&amp; \\\$6==&quot;G&quot;)||(\\\$5==&quot;G&quot; &amp;&amp; \\\$6==&quot;C&quot;)) print \\\$2, &quot;ambig&quot; ; else print \\\$2 ;}' &quot;<span class="math-container">${bfiles[1]
}" | grep ambig &gt; "$</span>{prefix}<span class="math-container">${chr_name}.step1b.snplist.txt"
plink1.9 --bfile "$</span>{prefix}<span class="math-container">${chr_name}.step1b" --exclude "$</span>{prefix}<span class="math-container">${chr_name}.step1b.snplist.txt" --make-bed --out "$</span>{prefix}${chr_name}.step1c&quot;
&quot;&quot;&quot;
}
</code></pre>

<plink><nextflow>
Nextflow: Runs without error, produce few but not all output files into output channel
# Nextflow: Runs without error, produce few but not all output files into output channel

(Asked by: username_0 on 2021-03-25)

I was trying to develop a pipeline using nextflow and plink. My code runs without any error, but got only two output files instead of 22 (one for each chromosome). Here is part my code block, any help?
process step5 {
publishDir params.publish, mode: 'copy'
input:
val(hrc) from HRC_1000G_check_bim
val(hrc_site) from hrc_sites_file
tuple val(prefix), val(chr_name), path(bfiles) from step4_frq
output:
tuple val("${prefix}"), val(chr_name), path("LOG-${prefix}${chr_name}.step4-HRC.txt") into step4_log
tuple val("${prefix}"), val(chr_name), path("Exclude-${prefix}${chr_name}.step4-HRC.txt") into step4_exclude
tuple val("${prefix}"), val(chr_name), path("Chromosome-${prefix}${chr_name}.step4-HRC.txt") into step4_chromosome
tuple val("${prefix}"), val(chr_name), path("Position-${prefix}${chr_name}.step4-HRC.txt") into step4_position
tuple val("${prefix}"), val(chr_name), path("Strand-Flip-${prefix}${chr_name}.step4-HRC.txt") into step4_strand
tuple val("${prefix}"), val(chr_name), path("Force-Allele1-${prefix}${chr_name}.step4-HRC.txt") into step4_allele
tuple val("${prefix}"), val(chr_name), path("ID-${prefix}${chr_name}.step4-HRC.txt") into step4_id
script:
"""
perl "${hrc}" -b "${prefix}${chr_name}.step4.bim" -f "${prefix}${chr_name}.step4.frq" -r "${hrc_site}" -h
"""
}

The output files are:
gsa-out/Chromosome-chr22.step4-HRC.txt     gsa-out/Force-Allele1-chr3.step4-HRC.txt  gsa-out/Position-chr22.step4-HRC.txt
gsa-out/Chromosome-chr3.step4-HRC.txt      gsa-out/ID-chr22.step4-HRC.txt            gsa-out/Position-chr3.step4-HRC.txt
gsa-out/Exclude-chr22.step4-HRC.txt        gsa-out/ID-chr3.step4-HRC.txt             gsa-out/Strand-Flip-chr22.step4-HRC.txt
gsa-out/Exclude-chr3.step4-HRC.txt         gsa-out/LOG-chr22.step4-HRC.txt           gsa-out/Strand-Flip-chr3.step4-HRC.txt
gsa-out/Force-Allele1-chr22.step4-HRC.txt  gsa-out/LOG-chr3.step4-HRC.txt

I am not sure why I am getting only 2, instead of 22 output files. With similar output channel previously generated 22 output files in the same script. For eaxmple:
process step1b {
publishDir params.publish, mode: 'copy'
input:
tuple val(prefix), val(chr_name), path(bfiles) from step1a_out
output:
tuple val("${prefix}"), val(chr_name), path("${prefix}${chr_name}.step1c.{bed,bim,fam}") into step1b_results
path("${prefix}${chr_name}.step1c.log") into step1b_logs
path("${prefix}${chr_name}.step1b.snplist.txt") into step1b_snplist
script:

"""
awk '{ if ((\\\$5=="T" && \\\$6=="A")||(\\\$5=="A" && \\\$6=="T")||(\\\$5=="C" && \\\$6=="G")||(\\\$5=="G" && \\\$6=="C")) print \\\$2, "ambig" ; else print \\\$2 ;}' "${bfiles[1]
}" | grep ambig > "${prefix}${chr_name}.step1b.snplist.txt"
plink1.9 --bfile "${prefix}${chr_name}.step1b" --exclude "${prefix}${chr_name}.step1b.snplist.txt" --make-bed --out "${prefix}${chr_name}.step1c"
"""
}

**Tags:** plink, nextflow

---

## 1 answers

### Answer by username_1 on 2021-03-25. Score: 1 | Accepted answer

I suspect your HRC_1000G_check_bim and hrc_sites_file channels are queue channels when they should be value channels. Please again see how multiple channels work. Also, third-party scripts, like your Perl script, would be better placed in a directory called bin/ at the root of your project repository. Nextflow adds this directory to your $PATH, so scripts in this directory will be automatically accessible in your pipeline without needing to localize them using a separate channel. This way also avoids having to specify an absolute path to invoke them. Just make sure your script is executable: chmod +x your_script.pl
hrc_sites = file( params.hrc_sites_file )

process step5 {

    input:
    tuple val(prefix), val(chr_name), path(bfiles) from step4_frq
    path hrc_sites

    """
    your_script.pl \\
        -b "${prefix}${chr_name}.step4.bim" \\
        -f "${prefix}${chr_name}.step4.frq" \\
        -r "${hrc_sites}" \\
        -h
    """
}

> **Comments:**
> **username_0**: Thank you `@Steve`. I just avoided using channel for those two files. I need to use them from absolute path, because other users also use this pipeline. However, it works now. One thing, I am using slurm executor with a config file, so far no problem with this. But I want to run the script in background (similar to sbatch submit, after submitting the job I will logout and the job will continue to run). How could I do this?  
> **username_0**: One thing, After finishing the jobs, transferring files to publish directory taking too much time. There are only 264 files but it is taking hours to transfer. Any solution for this?  
> **username_1**: No worries at all @zillurrahman. It's generally better to avoid specifying absolute paths in your process scripts. Avoiding absolute paths will mean that your process tasks can be run in isolation from each other and that your users could run your workflow on platforms like AWS Batch where there is no shared filesystem.  
> **username_1**: I don't have experience with Slurm, but you should be able to submit a job non-interactively using [sbatch](https://support.ceci-hpc.be/doc/_contents/QuickStart/SubmittingJobs/SlurmTutorial.html#creating-a-job) like you said. If it's submitted non-interactively, you can logout and the job running your workflow will continue on.  
> **username_1**: I'm not too sure about your last question; 264 files doesn't sound like a lot, but how much data are we talking? Could you be maxing out your network bandwidth? Are you writing to archival storage? Your sysadmin would be the right person to answer these sorts of questions and they might have some useful suggestions on how best run the workflow. For example, it might be better to write to a high performance disk array and then have a separate, but dependant, job copy these files to their final location.  
> **username_1**: You may also want to consider other publishing modes and parameterizing these, for example: `mode: params.publish_mode`. Consider also parameterizing the 'enabled' option so to let your users decide what outputs they want: `enabled: params.publish_everything || params.publish_step5`. HTH.  

---

