<p>Your <a href="https://www.nextflow.io/docs/stable/cache-and-resume.html#task-cache" rel="nofollow noreferrer">task cache</a> is being invalidated because your <a href="https://www.nextflow.io/docs/stable/cache-and-resume.html#non-deterministic-process-inputs" rel="nofollow noreferrer">process inputs are non-deterministic</a>. Like in the example, the solution is to <a href="https://www.nextflow.io/docs/stable/operator.html#join" rel="nofollow noreferrer">join</a> the two output channels before process invocation. Another way is to simply output the trimmed reads in a single channel, for example:</p>
<pre><code>process cut_adapt_trim {

    tag &quot;$replicateId&quot;

    publishDir '/Volumes/FrailtData/ATAC_Seq_downloaded/results', mode: 'copy'

    conda 'cutadapt'

    input:
    tuple val(replicateId), path(reads)
    
    output:
    tuple val(replicateId), path(&quot;<span class="math-container">${replicateId}.1.trimmed.fastq"), path("$</span>{replicateId}.2.trimmed.fastq&quot;)
    
    script:
    &quot;&quot;&quot;
    cutadapt \\
        -a TCGTCGGCAGCGTCAGATGTGTATAAGAGACAG \\
        -A GTCTCGTGGGCTCGGAGATGTGTATAAGAGACAG \\
        -o &quot;<span class="math-container">${replicateId}.1.trimmed.fastq" \\
        -p "$</span>{replicateId}.2.trimmed.fastq&quot; \\
        ${reads}
    &quot;&quot;&quot;
}
</code></pre>
<pre><code>process align_to_genome {

    tag &quot;$replicateId&quot;

    publishDir '/Volumes/FrailtData/ATAC_Seq_downloaded/results', mode: 'copy'

    conda 'bowtie2'

    input:
    tuple val(replicateId), path(fastq_1), path(fastq_2)
    path index

    output:
    tuple val(replicateId), path(&quot;${replicateId}.bam&quot;)

    script:
    def bwt_index = index[0].simpleName

    &quot;&quot;&quot;
    bowtie2 \\
        -p 64 \\
        -q \\
        -x &quot;<span class="math-container">${bwt_index}" \\
        -1 "$</span>{fastq_1}&quot; \\
        -2 &quot;<span class="math-container">${fastq_2}" |
    samtools view \\
        -b \\
        -o "$</span>{replicateId}.bam&quot; \\
        -
    &quot;&quot;&quot;
}
</code></pre>
<pre><code>workflow{

    reads_ch = Channel.fromFilePairs( params.reads )
    index_ch = Channel.fromPath( params.index )

    cut_adapt_trim( reads_ch )

    align_to_genome( cut_adapt_trim.out, index_ch )

    align_to_genome.out.view()

}
</code></pre>
<p>Run using:</p>
<pre><code>nextflow run run_nextflow_ATACseq.nf -with-conda -resume 
</code></pre>

CC BY-SA 4.0
2024-10-06T13:35:59.747Z
2024-10-06T13:45:56.897Z
2024-10-06T13:45:56.897Z
Steve
<p>You can achieve your desired checkpointing with some if statements and a user defined flag pretty easily. Something like:</p>
<pre><code>#!/usr/bin/env nextflow
nextflow.enable.dsl = 2

params.skip_trim = null 
params.reads = '/Volumes/FrailtData/ATAC_Seq_downloaded/PB_CD4plusCD45RAmin_D*_T72_*ADA_ATAC_{1,2}.fastq.gz'
params.index = '/Users/siddhaduio.no/Desktop/All_omics_tools/bowtie2-2.4.4-macos-x86_64/hg38_index/human_hg38_index.*.bt2'
params.trimmed_dir='01_trimmed'

process cut_adapt_trim {
   publishDir ${params.trimmed_dir}, mode: 'copy'

   input:
   tuple val(replicateId), path(reads)

   output:
   tuple val(replicateId), path(&quot;<span class="math-container">${replicateId}.1.trimmed.fastq"), emit: trimmed_pair_1
tuple val(replicateId), path("$</span>{replicateId}.2.trimmed.fastq&quot;), emit: trimmed_pair_2

   script:
   &quot;&quot;&quot;
   cutadapt -a TCGTCGGCAGCGTCAGATGTGTATAAGAGACAG -A GTCTCGTGGGCTCGGAGATGTGTATAAGAGACAG -o <span class="math-container">${replicateId}.1.trimmed.fastq -p $</span>{replicateId}.2.trimmed.fastq ${reads}
   &quot;&quot;&quot;
}


process align_to_genome {
   publishDir '/Volumes/FrailtData/ATAC_Seq_downloaded/results', mode: 'copy'

   input:
   path index
   tuple val(replicateId), path(&quot;/Volumes/FrailtData/ATAC_Seq_downloaded/results/<span class="math-container">${replicateId}.1.trimmed.fastq")
tuple val(replicateId), path("/Volumes/FrailtData/ATAC_Seq_downloaded/results/$</span>{replicateId}.2.trimmed.fastq&quot;)

   output:
   tuple val(replicateId), path(&quot;<span class="math-container">${replicateId}.sam"), emit:aligned_sam
tuple val(replicateId), path("$</span>{replicateId}.bam&quot;), emit:aligned_bam

   script:
   &quot;&quot;&quot;
   def bwt_index = index[0].simpleName


   /Users/siddhaduio.no/Desktop/All_omics_tools/bowtie2-2.4.4-macos-x86_64/bowtie2 -p 64 -q -x <span class="math-container">${bwt_index} -1 $</span>{replicateId}.1.trimmed.fastq -2 <span class="math-container">${replicateId}.2.trimmed.fastq -S $</span>{replicateId}.Aligned.sam 2&gt; <span class="math-container">${replicateId}.bowtie2.log | /Users/siddhaduio.no/Desktop/All_omics_tools/samtools-1.14/samtools view -S -b - &gt; $</span>{replicateId}.bam

   &quot;&quot;&quot;
}



workflow{
   index_ch= Channel.fromPath(params.index)
   if( params.skip_trim == false ) {
       reads_ch= Channel.fromFilePairs(params.reads)
       cut_adapt_trim(reads_ch)
       align_to_genome(index_ch,cut_adapt_trim.out)
   } else if( params.skip_trim ) {
       trim_ch= Channel.fromFilePairs(&quot;${params.trimmed_dir}/*_{1,2}.trimmed.fastq&quot;)
       align_to_genome(index_ch,trim_ch)
   }
}
</code></pre>
<p>Then to execute the skip you would use:</p>
<pre><code>nextflow run run_nextflow_ATACseq.nf -w /Volumes/FrailtData/ATAC_Seq_downloaded/results --skip_trim
</code></pre>
<p>Play around with the <code>trimmed_dir</code> variable, I don't know what you are mounting to. But using variables for a lot of the paths and options into the params section makes it easier to adjust in the future for other files.</p>
<p>There is another point I would like to address. Nextflow emits and inputs files in random order. So by splitting the R1 and R2 reads into separate channels, you are going to have a lot of mixed samples as your output. Make the output a single tuple: <code>tuple val(id), path(r1), path(r2), emit: trimmed_reads</code></p>

CC BY-SA 4.0
2024-10-07T20:48:28.443Z
Thank you again for the feedback. Very helpful!
siddhartha das
CC BY-SA 4.0
2024-10-06T10:36:07.337Z
2024-10-06T10:36:07.337Z
dthorbur
<p>I am trying to use nextflow to process some ATAC seq data. The data is paired end and the files are huge. So, I first trimmed the adapters and now I want to align them to the genome.
My script is shown below. Now, I want to align them using bowtie2. However, when I run the script, the trimming is being rerun again instead of starting at the alignment part.
Below is my call to nextflow at the command line</p>
<pre><code>nextflow run run_nextflow_ATACseq.nf -w /Volumes/FrailtData/ATAC_Seq_downloaded/results -resume
</code></pre>
<p>I was wondering what I am doing wrong such that the resume flag is not working?</p>
<p>Thanks for any suggestions!</p>
<pre><code>#!/usr/bin/env nextflow

nextflow.enable.dsl = 2


params.reads = '/Volumes/FrailtData/ATAC_Seq_downloaded/PB_CD4plusCD45RAmin_D*_T72_*ADA_ATAC_{1,2}.fastq.gz'
params.index = '/Users/siddhaduio.no/Desktop/All_omics_tools/bowtie2-2.4.4-macos-x86_64/hg38_index/human_hg38_index.*.bt2'



/*
* Trim atac seq reads to align to genome with cutadapt
*/



process cut_adapt_trim {


publishDir '/Volumes/FrailtData/ATAC_Seq_downloaded/results', mode: 'copy'

input:

tuple val(replicateId), path(reads)



output:

tuple val(replicateId), path(&quot;<span class="math-container">${replicateId}.1.trimmed.fastq"), emit: trimmed_pair_1
tuple val(replicateId), path("$</span>{replicateId}.2.trimmed.fastq&quot;), emit: trimmed_pair_2



script:
&quot;&quot;&quot;


cutadapt -a TCGTCGGCAGCGTCAGATGTGTATAAGAGACAG -A GTCTCGTGGGCTCGGAGATGTGTATAAGAGACAG -o <span class="math-container">${replicateId}.1.trimmed.fastq -p $</span>{replicateId}.2.trimmed.fastq ${reads}

&quot;&quot;&quot;


}


/*
* Align trimmed atac seq reads to genome
*/


process align_to_genome {

publishDir '/Volumes/FrailtData/ATAC_Seq_downloaded/results', mode: 'copy'


input:

path index

tuple val(replicateId), path(&quot;/Volumes/FrailtData/ATAC_Seq_downloaded/results/<span class="math-container">${replicateId}.1.trimmed.fastq")
tuple val(replicateId), path("/Volumes/FrailtData/ATAC_Seq_downloaded/results/$</span>{replicateId}.2.trimmed.fastq&quot;)


output:

tuple val(replicateId), path(&quot;<span class="math-container">${replicateId}.sam"), emit:aligned_sam
tuple val(replicateId), path("$</span>{replicateId}.bam&quot;), emit:aligned_bam


script:

&quot;&quot;&quot;
def bwt_index = index[0].simpleName


/Users/siddhaduio.no/Desktop/All_omics_tools/bowtie2-2.4.4-macos-x86_64/bowtie2 -p 64 -q -x <span class="math-container">${bwt_index} -1 $</span>{replicateId}.1.trimmed.fastq -2 <span class="math-container">${replicateId}.2.trimmed.fastq -S $</span>{replicateId}.Aligned.sam 2&gt; <span class="math-container">${replicateId}.bowtie2.log | /Users/siddhaduio.no/Desktop/All_omics_tools/samtools-1.14/samtools view -S -b - &gt; $</span>{replicateId}.bam

&quot;&quot;&quot;

}



workflow{

reads_ch= Channel.fromFilePairs(params.reads)
index_ch= Channel.fromPath(params.index)
Channel.fromFilePairs(params.reads).view()
index_ch.view()
cut_adapt_trim(reads_ch)
align_to_genome(index_ch,cut_adapt_trim.out)

}
<span class="math-container">```</span>
</code></pre>

<nextflow>
How do I start a nextflow run from last executed command
# How do I start a nextflow run from last executed command

(Asked by: username_0 on 2024-10-06)

I am trying to use nextflow to process some ATAC seq data. The data is paired end and the files are huge. So, I first trimmed the adapters and now I want to align them to the genome.
My script is shown below. Now, I want to align them using bowtie2. However, when I run the script, the trimming is being rerun again instead of starting at the alignment part.
Below is my call to nextflow at the command line
nextflow run run_nextflow_ATACseq.nf -w /Volumes/FrailtData/ATAC_Seq_downloaded/results -resume

I was wondering what I am doing wrong such that the resume flag is not working?
Thanks for any suggestions!
#!/usr/bin/env nextflow

nextflow.enable.dsl = 2


params.reads = '/Volumes/FrailtData/ATAC_Seq_downloaded/PB_CD4plusCD45RAmin_D*_T72_*ADA_ATAC_{1,2}.fastq.gz'
params.index = '/Users/siddhaduio.no/Desktop/All_omics_tools/bowtie2-2.4.4-macos-x86_64/hg38_index/human_hg38_index.*.bt2'



/*
* Trim atac seq reads to align to genome with cutadapt
*/



process cut_adapt_trim {


publishDir '/Volumes/FrailtData/ATAC_Seq_downloaded/results', mode: 'copy'

input:

tuple val(replicateId), path(reads)



output:

tuple val(replicateId), path("${replicateId}.1.trimmed.fastq"), emit: trimmed_pair_1
tuple val(replicateId), path("${replicateId}.2.trimmed.fastq"), emit: trimmed_pair_2



script:
"""


cutadapt -a TCGTCGGCAGCGTCAGATGTGTATAAGAGACAG -A GTCTCGTGGGCTCGGAGATGTGTATAAGAGACAG -o ${replicateId}.1.trimmed.fastq -p ${replicateId}.2.trimmed.fastq ${reads}

"""


}


/*
* Align trimmed atac seq reads to genome
*/


process align_to_genome {

publishDir '/Volumes/FrailtData/ATAC_Seq_downloaded/results', mode: 'copy'


input:

path index

tuple val(replicateId), path("/Volumes/FrailtData/ATAC_Seq_downloaded/results/${replicateId}.1.trimmed.fastq")
tuple val(replicateId), path("/Volumes/FrailtData/ATAC_Seq_downloaded/results/${replicateId}.2.trimmed.fastq")


output:

tuple val(replicateId), path("${replicateId}.sam"), emit:aligned_sam
tuple val(replicateId), path("${replicateId}.bam"), emit:aligned_bam


script:

"""
def bwt_index = index[0].simpleName


/Users/siddhaduio.no/Desktop/All_omics_tools/bowtie2-2.4.4-macos-x86_64/bowtie2 -p 64 -q -x ${bwt_index} -1 ${replicateId}.1.trimmed.fastq -2 ${replicateId}.2.trimmed.fastq -S ${replicateId}.Aligned.sam 2> ${replicateId}.bowtie2.log | /Users/siddhaduio.no/Desktop/All_omics_tools/samtools-1.14/samtools view -S -b - > ${replicateId}.bam

"""

}



workflow{

reads_ch= Channel.fromFilePairs(params.reads)
index_ch= Channel.fromPath(params.index)
Channel.fromFilePairs(params.reads).view()
index_ch.view()
cut_adapt_trim(reads_ch)
align_to_genome(index_ch,cut_adapt_trim.out)

}
```

**Tags:** nextflow

---

## 2 answers

### Answer by username_1 on 2024-10-06. Score: 2

Your task cache is being invalidated because your process inputs are non-deterministic. Like in the example, the solution is to join the two output channels before process invocation. Another way is to simply output the trimmed reads in a single channel, for example:
process cut_adapt_trim {

    tag "$replicateId"

    publishDir '/Volumes/FrailtData/ATAC_Seq_downloaded/results', mode: 'copy'

    conda 'cutadapt'

    input:
    tuple val(replicateId), path(reads)
    
    output:
    tuple val(replicateId), path("${replicateId}.1.trimmed.fastq"), path("${replicateId}.2.trimmed.fastq")
    
    script:
    """
    cutadapt \\
        -a TCGTCGGCAGCGTCAGATGTGTATAAGAGACAG \\
        -A GTCTCGTGGGCTCGGAGATGTGTATAAGAGACAG \\
        -o "${replicateId}.1.trimmed.fastq" \\
        -p "${replicateId}.2.trimmed.fastq" \\
        ${reads}
    """
}

process align_to_genome {

    tag "$replicateId"

    publishDir '/Volumes/FrailtData/ATAC_Seq_downloaded/results', mode: 'copy'

    conda 'bowtie2'

    input:
    tuple val(replicateId), path(fastq_1), path(fastq_2)
    path index

    output:
    tuple val(replicateId), path("${replicateId}.bam")

    script:
    def bwt_index = index[0].simpleName

    """
    bowtie2 \\
        -p 64 \\
        -q \\
        -x "${bwt_index}" \\
        -1 "${fastq_1}" \\
        -2 "${fastq_2}" |
    samtools view \\
        -b \\
        -o "${replicateId}.bam" \\
        -
    """
}

workflow{

    reads_ch = Channel.fromFilePairs( params.reads )
    index_ch = Channel.fromPath( params.index )

    cut_adapt_trim( reads_ch )

    align_to_genome( cut_adapt_trim.out, index_ch )

    align_to_genome.out.view()

}

Run using:
nextflow run run_nextflow_ATACseq.nf -with-conda -resume


---

### Answer by username_2 on 2024-10-06. Score: 2

You can achieve your desired checkpointing with some if statements and a user defined flag pretty easily. Something like:
#!/usr/bin/env nextflow
nextflow.enable.dsl = 2

params.skip_trim = null 
params.reads = '/Volumes/FrailtData/ATAC_Seq_downloaded/PB_CD4plusCD45RAmin_D*_T72_*ADA_ATAC_{1,2}.fastq.gz'
params.index = '/Users/siddhaduio.no/Desktop/All_omics_tools/bowtie2-2.4.4-macos-x86_64/hg38_index/human_hg38_index.*.bt2'
params.trimmed_dir='01_trimmed'

process cut_adapt_trim {
   publishDir ${params.trimmed_dir}, mode: 'copy'

   input:
   tuple val(replicateId), path(reads)

   output:
   tuple val(replicateId), path("${replicateId}.1.trimmed.fastq"), emit: trimmed_pair_1
tuple val(replicateId), path("${replicateId}.2.trimmed.fastq"), emit: trimmed_pair_2

   script:
   """
   cutadapt -a TCGTCGGCAGCGTCAGATGTGTATAAGAGACAG -A GTCTCGTGGGCTCGGAGATGTGTATAAGAGACAG -o ${replicateId}.1.trimmed.fastq -p ${replicateId}.2.trimmed.fastq ${reads}
   """
}


process align_to_genome {
   publishDir '/Volumes/FrailtData/ATAC_Seq_downloaded/results', mode: 'copy'

   input:
   path index
   tuple val(replicateId), path("/Volumes/FrailtData/ATAC_Seq_downloaded/results/${replicateId}.1.trimmed.fastq")
tuple val(replicateId), path("/Volumes/FrailtData/ATAC_Seq_downloaded/results/${replicateId}.2.trimmed.fastq")

   output:
   tuple val(replicateId), path("${replicateId}.sam"), emit:aligned_sam
tuple val(replicateId), path("${replicateId}.bam"), emit:aligned_bam

   script:
   """
   def bwt_index = index[0].simpleName


   /Users/siddhaduio.no/Desktop/All_omics_tools/bowtie2-2.4.4-macos-x86_64/bowtie2 -p 64 -q -x ${bwt_index} -1 ${replicateId}.1.trimmed.fastq -2 ${replicateId}.2.trimmed.fastq -S ${replicateId}.Aligned.sam 2> ${replicateId}.bowtie2.log | /Users/siddhaduio.no/Desktop/All_omics_tools/samtools-1.14/samtools view -S -b - > ${replicateId}.bam

   """
}



workflow{
   index_ch= Channel.fromPath(params.index)
   if( params.skip_trim == false ) {
       reads_ch= Channel.fromFilePairs(params.reads)
       cut_adapt_trim(reads_ch)
       align_to_genome(index_ch,cut_adapt_trim.out)
   } else if( params.skip_trim ) {
       trim_ch= Channel.fromFilePairs("${params.trimmed_dir}/*_{1,2}.trimmed.fastq")
       align_to_genome(index_ch,trim_ch)
   }
}

Then to execute the skip you would use:
nextflow run run_nextflow_ATACseq.nf -w /Volumes/FrailtData/ATAC_Seq_downloaded/results --skip_trim

Play around with the trimmed_dir variable, I don't know what you are mounting to. But using variables for a lot of the paths and options into the params section makes it easier to adjust in the future for other files.
There is another point I would like to address. Nextflow emits and inputs files in random order. So by splitting the R1 and R2 reads into separate channels, you are going to have a lot of mixed samples as your output. Make the output a single tuple: tuple val(id), path(r1), path(r2), emit: trimmed_reads

> **Comments:**
> **username_0**: Thank you again for the feedback. Very helpful!  

---

