<p>Unfortunately, this remains an open issue with Nextflow: <a href="https://github.com/nextflow-io/nextflow/issues/452" rel="nofollow noreferrer">https://github.com/nextflow-io/nextflow/issues/452</a></p>
<p>However, writing large amounts of uncompressed data to disk is usually best avoided. If it can't be (e.g. maybe the file needs to be read multiple times), you'd need to merge the two processes so that the writing of the temporary file and the running of the tool is done in a single process. This would then let you remove the temporary input file as soon as it's no longer required.</p>
<p>Another way would be to pipe the uncompressed and FASTA formatted reads directly into your tool. Not all tools support this functionality obviously, but <a href="https://github.com/ridgelab/Kmer-SSR" rel="nofollow noreferrer">Kmer-SSR</a> <em>can</em> indeed receive input from STDIN, which completely obviates the need to write any temporary files. Note that by default, Kmer-SSR writes also to STDOUT and this lets you write a compressed output file:</p>
<pre><code>process kmer_ssr {

    cpus 4

    input:
    tuple val(sample), path(fastq)

    output:
    tuple val(sample), path(&quot;${sample}.tsv.gz&quot;)

    &quot;&quot;&quot;
    zcat -f &quot;<span class="math-container">${fastq}" \\
        | sed -n '1~4s/^@/&gt;/p; 2~4p' \\
        | kmer-ssr -t $</span>{task.cpus} \\
        | gzip \\
        &gt; &quot;${sample}.tsv.gz&quot;
    &quot;&quot;&quot;
}
</code></pre>
<p>Since the above uses a pipeline, make sure to add the <code>pipefail</code> to your command. One way is to simply apply it to all processes by adding the following to your <code>nextflow.config</code>:</p>
<pre><code>process {

  shell = [ '/bin/bash', '-euo', 'pipefail' ]
}
</code></pre>
<p>From the <code>bash</code> man page:</p>
<blockquote>
<p>pipefail</p>
<p>If set, the return value of a pipeline is the value of the last
(rightmost) command to exit with a non-zero status, or zero if all
commands in the pipeline exit successfully. This option is disabled by
default.</p>
</blockquote>

CC BY-SA 4.0
2023-02-12T03:25:57.977Z
2023-02-12T03:25:57.977Z
Steve
<p>In my pipeline nextflow, I create a channel with channel.fromSRA and the channel contains lots of heavy files.fastq.gz. Then I have a first process to unzip files and transform them in files.fasta and a second process to count kmers. Is it possible to remove files.gz after first process and remove files.fasta after the second process. Can someone explain me how to do? They are too heavy to keep them in the work directory. Thanks.</p>
<p>In my pipeline I use kmer-ssr to count kmers present on my genome sample.</p>
<p>First I create a reads_ch = Channel.fromSRA</p>
<p>In this channel there are 12 files (. fastq.gz) of 20 Gb each one.</p>
<p>Then the first process unzips them and transforms them in files.fasta.</p>
<p>Now in my work directory there are 12 more files of 80 Gb each one. So the work directory is now 1200 Gb.</p>
<p>The second process counts kmer present in reads and creates 12 files of 100 Gb each one. The work directory will be 2200 Gb.</p>
<p>The work directory is too heavy. I want to know how I can suppress the current file. fastq.gz after first process</p>

CC BY-SA 4.0
2023-02-11T13:26:17.107Z
Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.
Community
CC BY-SA 4.0
2023-02-11T13:31:28.373Z
Certainly this will be doable. Could you provide the code? Or else sufficient amounts of code to describe the nested loop? I will provide a link below as typical example of a `NextFlow` question.
M__
CC BY-SA 4.0
2023-02-11T13:33:01.407Z
So this is yesterdays question [here](https://bioinformatics.stackexchange.com/questions/20536/nextflow-tuple-from-csv-cols/20537#20537) as an example of a standard `NextFlow` question on the site.
M__
CC BY-SA 4.0
2023-04-13T00:59:55.910Z
Look into the "temporary" attribute of the inputs that should help you in the right direction.
raygozag
<fasta><fastq><nextflow><k-mer><workflow-management>
Remove files after process is terminated
# Remove files after process is terminated

(Asked by: username_0 on 2023-02-11)

In my pipeline nextflow, I create a channel with channel.fromSRA and the channel contains lots of heavy files.fastq.gz. Then I have a first process to unzip files and transform them in files.fasta and a second process to count kmers. Is it possible to remove files.gz after first process and remove files.fasta after the second process. Can someone explain me how to do? They are too heavy to keep them in the work directory. Thanks.

In my pipeline I use kmer-ssr to count kmers present on my genome sample.

First I create a reads\_ch = Channel.fromSRA

In this channel there are 12 files (. fastq.gz) of 20 Gb each one.

Then the first process unzips them and transforms them in files.fasta.

Now in my work directory there are 12 more files of 80 Gb each one. So the work directory is now 1200 Gb.

The second process counts kmer present in reads and creates 12 files of 100 Gb each one. The work directory will be 2200 Gb.

The work directory is too heavy. I want to know how I can suppress the current file. fastq.gz after first process

**Tags:** fasta, fastq, nextflow, k-mer, workflow-management
> **Comments:**
> **username_1**: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.  
> **username_2**: Certainly this will be doable. Could you provide the code? Or else sufficient amounts of code to describe the nested loop? I will provide a link below as typical example of a `NextFlow` question.  
> **username_2**: So this is yesterdays question [here](https://bioinformatics.stackexchange.com/questions/20536/nextflow-tuple-from-csv-cols/20537#20537) as an example of a standard `NextFlow` question on the site.  
> **username_3**: Look into the "temporary" attribute of the inputs that should help you in the right direction.  

---

## 1 answers

### Answer by username_4 on 2023-02-12. Score: 0

Unfortunately, this remains an open issue with Nextflow: https://github.com/nextflow-io/nextflow/issues/452
However, writing large amounts of uncompressed data to disk is usually best avoided. If it can't be (e.g. maybe the file needs to be read multiple times), you'd need to merge the two processes so that the writing of the temporary file and the running of the tool is done in a single process. This would then let you remove the temporary input file as soon as it's no longer required.
Another way would be to pipe the uncompressed and FASTA formatted reads directly into your tool. Not all tools support this functionality obviously, but Kmer-SSR can indeed receive input from STDIN, which completely obviates the need to write any temporary files. Note that by default, Kmer-SSR writes also to STDOUT and this lets you write a compressed output file:
process kmer_ssr {

    cpus 4

    input:
    tuple val(sample), path(fastq)

    output:
    tuple val(sample), path("${sample}.tsv.gz")

    """
    zcat -f "${fastq}" \\
        | sed -n '1~4s/^@/>/p; 2~4p' \\
        | kmer-ssr -t ${task.cpus} \\
        | gzip \\
        > "${sample}.tsv.gz"
    """
}

Since the above uses a pipeline, make sure to add the pipefail to your command. One way is to simply apply it to all processes by adding the following to your nextflow.config:
process {

  shell = [ '/bin/bash', '-euo', 'pipefail' ]
}

From the bash man page:

pipefail
If set, the return value of a pipeline is the value of the last
(rightmost) command to exit with a non-zero status, or zero if all
commands in the pipeline exit successfully. This option is disabled by
default.


---

