<p>How do you know your workflow actually finishes successfully? One way to check the workflow termination status is to use the <a href="https://www.nextflow.io/docs/latest/metadata.html#completion-handler" rel="nofollow noreferrer"><code>workflow.onComplete</code></a> and <a href="https://www.nextflow.io/docs/latest/metadata.html#error-handler" rel="nofollow noreferrer"><code>workflow.onError</code></a> event handlers. Consider resuming your workflow with something like the following appended to your script:</p>
<pre><code>workflow.onComplete {

  log.info &quot;Workflow completed at: <span class="math-container">${workflow.complete}"
  log.info "Time taken: $</span>{workflow.duration}&quot;
  log.info &quot;Execution status: ${workflow.success ? 'success' : 'failed'}&quot;
}

workflow.onError {

  log.info &quot;Execution halted: ${workflow.errorMessage}&quot;
}
</code></pre>
<p>Note that the workflow event handlers may also be defined <a href="https://www.nextflow.io/docs/latest/metadata.html#decoupling-metadata" rel="nofollow noreferrer">inside</a> your <code>nextflow.config</code> if preferred.</p>

CC BY-SA 4.0
2022-02-28T14:45:01.337Z
Thanks, @Steve. I added an example to my post. Seems that one way or another, the example I show above should have completed because `Nextflow` has nothing left to do.
Mark Ebbert
CC BY-SA 4.0
2022-03-01T00:14:05.823Z
@MarkEbbert It's hard to say what the problem is here exactly without the nextflow.log or a reproducible example. But just because all of the tasks have completed (regardless of the exit status) doesn't necessarily mean Nextflow has nothing left to do. For example, the publishDir directive works asynchronously and can be used (as you know) to 'copy' task outputs. If these are large and there are many of these, Nextflow will hang around copying files from the queue until well after the last task has completed.
Steve
CC BY-SA 4.0
2022-03-01T19:07:17.953Z
Thanks, @Steve. I just added part of the `.nextflow.log` file. Looks like `Nextflow` thinks it's hung, too? I haven't tested whether this is reproducible, by this is the code for my related question. I included a link to it above. This one may just remain a mystery.
Mark Ebbert
CC BY-SA 4.0
2022-03-01T19:25:06.070Z
I can't edit my above comment, but should read "..., but this is from the code..."
Mark Ebbert
CC BY-SA 4.0
2022-03-02T02:02:32.290Z
@MarkEbbert Correct - this looks like a bug. It might have something to do with the errorStrategy set to 'ignore' for all processes, potentially. Basically, you need to hit [this condition](https://github.com/nextflow-io/nextflow/blob/v21.10.6/modules/nextflow/src/main/groovy/nextflow/processor/TaskPollingMonitor.groovy#L412) to break out of the poll loop. From the partial log you've posted, it looks like the runningQueue is empty, but I don't know about the pendingQueue.size() or if session.isTerminated().
Steve
CC BY-SA 4.0
2022-03-02T02:02:56.337Z
@MarkEbbert You might like to open an [issue](https://github.com/nextflow-io/nextflow/issues) with the full log attached.
Steve
CC BY-SA 4.0
2022-03-02T20:57:08.813Z
Thanks, @Steve. Submitted: https://github.com/nextflow-io/nextflow/issues/2693. I'll report back if I learn something.
Mark Ebbert
CC BY-SA 4.0
2022-02-27T14:00:35.317Z
2022-02-27T14:00:35.317Z
Steve
<h2>Bug fixed</h2>
<p>This issue appears to have been fixed in latest <code>Nextflow</code>: <a href="https://github.com/nextflow-io/nextflow/issues/2693" rel="nofollow noreferrer">https://github.com/nextflow-io/nextflow/issues/2693</a></p>
<hr />
<p>I've been using <code>Nextflow</code> for various workflows on a cluster using the <code>SLURM</code> scheduler. I've noticed that the 'parent' job does not shut down when the <code>Nextflow</code> workflow finishes successfully. I can cancel the job, but then <code>Nextflow</code> doesn't get a chance to clean things up and also doesn't write out the <code>trace</code>, <code>report</code>, etc. The parent job does end properly if the workflow ends with an error.</p>
<p><strong>Question</strong>: Why doesn't the parent <code>Nextflow</code> job end upon successful completion?</p>
<p>I saw <a href="https://github.com/nextflow-io/nextflow/issues/968" rel="nofollow noreferrer">a post on the <code>Nextflow</code> GitHub</a> where the original poster describes various acrobatics to get the parent job to exit cleanly, but this is something that <code>Nextflow</code> should handle.</p>
<p>Here's my <code>SLURM</code> submission script that I submit using <code>sbatch run_workflow.slurm</code></p>
<p><strong>run_workflow.slurm</strong></p>
<pre><code>#!/bin/bash
#SBATCH --time=60:15:00                             # Time limit for the job (REQUIRED).
#SBATCH --job-name=NF_Parent_Identify_D&amp;C_regions   # Job name
#SBATCH --ntasks=1                                  # Number of cores for the job. Same as SBATCH
#SBATCH --mem=5G                                    # Total memory requested
#SBATCH --partition=&lt;partition&gt;                     # Partition/queue to run the job in. (REQUIRED
#SBATCH -e slurm-%j.err                             # Error file for this job.
#SBATCH -o slurm-%j.out                             # Output file for this job.
#SBATCH -A &lt;account&gt;                                # Project allocation account name (REQUIRED)

module load ccs/java/jdk1.8.0_202
nextflow run identify_dark_and_camouflaged_regions.nf \
    -with-report \
    -with-trace \
    -with-dag idcr-workflow.png \
    -with-timeline \
    -resume
</code></pre>
<h1>Update 1</h1>
<p>I'm adding example <code>Nextflow</code> output from a different workflow (I don't have the <code>SLURM</code> output from the one I mention above) where the workflow has completed, but the parent job is still running. In this case, there were errors, but I added <code>errorStrategy 'ignore'</code> to all of the processes listed:</p>
<pre><code>executor &gt;  slurm (876)
[06/c5672b] process &gt; samtools_csort_proc (A-CUHS... [100%] 304 of 304, faile...
[45/26a046] process &gt; Lsamtools_csort_proc (A-CUH... [100%] 304 of 304, faile...
[b5/0f2d7c] process &gt; sambamba_csort_proc (A-CUHS... [100%] 304 of 304, faile...
</code></pre>
<p>So, regardless of whether the workflow completed <code>successfully</code>, <code>Nextflow</code> has nothing left to do, but it's still running.</p>
<h1>Update 2 (adding partial <code>.nextflow.log</code> file)</h1>
<p>Per @Steve's point, I checked the <code>.nextflow.log</code> file and am including part of the file, here. In this case, I don't think <code>Nextflow</code> has any tasks related to <code>publishDir</code> because I'm not using that in this workflow. The job that's still running is from the <code>Nextflow</code> workflow in my GitHub repository for my answer posted in my related question <a href="https://bioinformatics.stackexchange.com/questions/18538/samtools-sort-most-efficient-memory-and-thread-settings-for-many-samples-on-a-c">Samtools sort: most efficient memory and thread settings for many samples on a cluster</a>. Whether it's reproducible, I haven't checked. Not sure how common this issue is.</p>
<p>In either case, my interpretation of the <code>.nextflow.log</code> file below is that <code>Nextflow</code> thinks it's hung, too. I just don't know why, or how to exit cleanly.</p>
<p><strong>Partial view of <code>.nextflow.log</code></strong></p>
<pre><code>...

Feb-28 05:29:04.607 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor slurm &gt; tasks to be completed: 1 -- submitted tasks are shown below
~&gt; TaskHandler[jobId: 662274; id: 876; name: sambamba_csort_proc (A-CUHS-CU003128-BL-COL-49696BL1.unsorted;CPUs:2;MEM_PER_THREAD:100GB); status: RUNNING; exit: -; error: -; workDir: /path/to/Samtools_sort_optimization_test/SORT_COMPARISON/work/b5/0f2d7c5d7f81a6e62dedb08bf6694a started: 1646038174334; exited: -; ]
Feb-28 05:30:54.613 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed &gt; TaskHandler[jobId: 662274; id: 876; name: sambamba_csort_proc (A-CUHS-CU003128-BL-COL-49696BL1.unsorted;CPUs:2;MEM_PER_THREAD:100GB); status: COMPLETED; exit: 0; error: -; workDir: /path/to/Samtools_sort_optimization_test/SORT_COMPARISON/work/b5/0f2d7c5d7f81a6e62dedb08bf6694a started: 1646038174334; exited: 2022-02-28T10:30:53.516015Z; ]
Feb-28 05:34:04.641 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled
Feb-28 05:39:04.647 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled
Feb-28 05:44:04.653 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled
Feb-28 05:49:04.658 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled
Feb-28 05:54:04.662 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled

&lt; above line repeats hundreds of times &gt;

...
</code></pre>

CC BY-SA 4.0
2022-05-17T05:21:49.237Z
Can you please create an answer with your accepted solution, so that the StackExchange algorithms don't confuse this with an unanswered question?
gringer
<nextflow><slurm>
Nextflow scheduler job doesn't end upon successful workflow completion
# Nextflow scheduler job doesn't end upon successful workflow completion

(Asked by: username_0 on 2022-02-26)

Bug fixed
This issue appears to have been fixed in latest Nextflow: https://github.com/nextflow-io/nextflow/issues/2693

I've been using Nextflow for various workflows on a cluster using the SLURM scheduler. I've noticed that the 'parent' job does not shut down when the Nextflow workflow finishes successfully. I can cancel the job, but then Nextflow doesn't get a chance to clean things up and also doesn't write out the trace, report, etc. The parent job does end properly if the workflow ends with an error.
Question: Why doesn't the parent Nextflow job end upon successful completion?
I saw a post on the Nextflow GitHub where the original poster describes various acrobatics to get the parent job to exit cleanly, but this is something that Nextflow should handle.
Here's my SLURM submission script that I submit using sbatch run_workflow.slurm
run_workflow.slurm
#!/bin/bash
#SBATCH --time=60:15:00                             # Time limit for the job (REQUIRED).
#SBATCH --job-name=NF_Parent_Identify_D&C_regions   # Job name
#SBATCH --ntasks=1                                  # Number of cores for the job. Same as SBATCH
#SBATCH --mem=5G                                    # Total memory requested
#SBATCH --partition=                     # Partition/queue to run the job in. (REQUIRED
#SBATCH -e slurm-%j.err                             # Error file for this job.
#SBATCH -o slurm-%j.out                             # Output file for this job.
#SBATCH -A                                 # Project allocation account name (REQUIRED)

module load ccs/java/jdk1.8.0_202
nextflow run identify_dark_and_camouflaged_regions.nf \
    -with-report \
    -with-trace \
    -with-dag idcr-workflow.png \
    -with-timeline \
    -resume

Update 1
I'm adding example Nextflow output from a different workflow (I don't have the SLURM output from the one I mention above) where the workflow has completed, but the parent job is still running. In this case, there were errors, but I added errorStrategy 'ignore' to all of the processes listed:
executor >  slurm (876)
[06/c5672b] process > samtools_csort_proc (A-CUHS... [100%] 304 of 304, faile...
[45/26a046] process > Lsamtools_csort_proc (A-CUH... [100%] 304 of 304, faile...
[b5/0f2d7c] process > sambamba_csort_proc (A-CUHS... [100%] 304 of 304, faile...

So, regardless of whether the workflow completed successfully, Nextflow has nothing left to do, but it's still running.
Update 2 (adding partial .nextflow.log file)
Per @Steve's point, I checked the .nextflow.log file and am including part of the file, here. In this case, I don't think Nextflow has any tasks related to publishDir because I'm not using that in this workflow. The job that's still running is from the Nextflow workflow in my GitHub repository for my answer posted in my related question Samtools sort: most efficient memory and thread settings for many samples on a cluster. Whether it's reproducible, I haven't checked. Not sure how common this issue is.
In either case, my interpretation of the .nextflow.log file below is that Nextflow thinks it's hung, too. I just don't know why, or how to exit cleanly.
Partial view of .nextflow.log
...

Feb-28 05:29:04.607 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor slurm > tasks to be completed: 1 -- submitted tasks are shown below
~> TaskHandler[jobId: 662274; id: 876; name: sambamba_csort_proc (A-CUHS-CU003128-BL-COL-49696BL1.unsorted;CPUs:2;MEM_PER_THREAD:100GB); status: RUNNING; exit: -; error: -; workDir: /path/to/Samtools_sort_optimization_test/SORT_COMPARISON/work/b5/0f2d7c5d7f81a6e62dedb08bf6694a started: 1646038174334; exited: -; ]
Feb-28 05:30:54.613 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[jobId: 662274; id: 876; name: sambamba_csort_proc (A-CUHS-CU003128-BL-COL-49696BL1.unsorted;CPUs:2;MEM_PER_THREAD:100GB); status: COMPLETED; exit: 0; error: -; workDir: /path/to/Samtools_sort_optimization_test/SORT_COMPARISON/work/b5/0f2d7c5d7f81a6e62dedb08bf6694a started: 1646038174334; exited: 2022-02-28T10:30:53.516015Z; ]
Feb-28 05:34:04.641 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled
Feb-28 05:39:04.647 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled
Feb-28 05:44:04.653 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled
Feb-28 05:49:04.658 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled
Feb-28 05:54:04.662 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- Execution may be stalled

< above line repeats hundreds of times >

...

**Tags:** nextflow, slurm
> **Comments:**
> **username_1**: Can you please create an answer with your accepted solution, so that the StackExchange algorithms don't confuse this with an unanswered question?  

---

## 1 answers

### Answer by username_2 on 2022-02-27. Score: 2

How do you know your workflow actually finishes successfully? One way to check the workflow termination status is to use the workflow.onComplete and workflow.onError event handlers. Consider resuming your workflow with something like the following appended to your script:
workflow.onComplete {

  log.info "Workflow completed at: ${workflow.complete}"
  log.info "Time taken: ${workflow.duration}"
  log.info "Execution status: ${workflow.success ? 'success' : 'failed'}"
}

workflow.onError {

  log.info "Execution halted: ${workflow.errorMessage}"
}

Note that the workflow event handlers may also be defined inside your nextflow.config if preferred.

> **Comments:**
> **username_0**: Thanks, @Steve. I added an example to my post. Seems that one way or another, the example I show above should have completed because `Nextflow` has nothing left to do.  
> **username_2**: @MarkEbbert It's hard to say what the problem is here exactly without the nextflow.log or a reproducible example. But just because all of the tasks have completed (regardless of the exit status) doesn't necessarily mean Nextflow has nothing left to do. For example, the publishDir directive works asynchronously and can be used (as you know) to 'copy' task outputs. If these are large and there are many of these, Nextflow will hang around copying files from the queue until well after the last task has completed.  
> **username_0**: Thanks, @Steve. I just added part of the `.nextflow.log` file. Looks like `Nextflow` thinks it's hung, too? I haven't tested whether this is reproducible, by this is the code for my related question. I included a link to it above. This one may just remain a mystery.  
> **username_0**: I can't edit my above comment, but should read "..., but this is from the code..."  
> **username_2**: @MarkEbbert Correct - this looks like a bug. It might have something to do with the errorStrategy set to 'ignore' for all processes, potentially. Basically, you need to hit [this condition](https://github.com/nextflow-io/nextflow/blob/v21.10.6/modules/nextflow/src/main/groovy/nextflow/processor/TaskPollingMonitor.groovy#L412) to break out of the poll loop. From the partial log you've posted, it looks like the runningQueue is empty, but I don't know about the pendingQueue.size() or if session.isTerminated().  
> **username_2**: @MarkEbbert You might like to open an [issue](https://github.com/nextflow-io/nextflow/issues) with the full log attached.  
> **username_0**: Thanks, @Steve. Submitted: https://github.com/nextflow-io/nextflow/issues/2693. I'll report back if I learn something.  

---

