<p>Optional outputs should work in WDL 1.0. Here is an example that runs without issue through cromwell (version 74).</p>
<p>Note that for the second call in the scatter the input file needs to be converted to from an optional to an non optional value. Since this only gets run if this value is defined, this conversion can safely be done using the <code>select_first</code> function (which returns the first defined value in a given list of optionals).</p>
<pre><code>version 1.0

workflow test {
    input {
        File filelist = &quot;files.txt&quot;
    }
    Array[String] files = read_lines(filelist)

    scatter (file in  files) {
         call blah as b1 {input: in=file}
         if (defined(b1.out)) {
             call blah as b2 {input: in=select_first([b1.out])}
         }
    }
}


task blah {
    input {
        File in
    }
    String outPath = basename(in,'.txt') + '_blah.txt'

    command {
        if $(grep -q &quot;yay&quot; ~{in})
          then
             echo yay &gt; ~{outPath}
        fi
    }

    output {
        File? out = outPath
    }
}
</code></pre>
<p>files.txt looks like this:</p>
<pre><code>test1.txt
test2.txt
</code></pre>
<p>With test1.txt containing <code>yay</code> and test2.txt containing <code>nay</code>.</p>

CC BY-SA 4.0
2022-05-05T09:45:47.247Z
2022-05-05T09:45:47.247Z
DavyCats
<p>I'm trying to adapt an existing workflow to make it more robust to failures. Below is a simplified example of the workflow.</p>
<pre><code>workflow main {
   File filelist=&quot;list_of_input_files.txt&quot;
   Array[String] files=read_lines(filelist)
   scatter(file in files) {
      call task1 { input: in=file }
      if(defined(task1.out)) { call task2 { input: in=task1.out } }
      if(defined(task2.out)) { call task2 { input: in=task2.out }
   }
}

task task1 {
   File in
   String out1 = basename(in,'.txt') + '_1.txt'
   command { if(...) { ./script1.sh <span class="math-container">${in} &gt; $</span>{out1} } }
   output { File? out = out1 }
   runtime { ... }
}

task task2 {
   File in
   String out2 = basename(in,'.txt') + '_2.txt'
   command { if(...) {./script2.sh <span class="math-container">${in} &gt; $</span>{out2} } }
   output { File? out = out2 }
   runtime { ... }
}

task task3 {
   File in
   String out3 = basename(in,'.txt') + '_3.txt'
   command { if(...) {./script3.sh <span class="math-container">${in} &gt; $</span>{out3} } }
   output { File? out = out3 }
   runtime {...}
}
</code></pre>
<p>with my list_of_input_files.txt looking like</p>
<pre><code>firstfile.txt
secondfile.txt
thirdfile.txt
...
</code></pre>
<p>Obviously this is oversimplified, but I am trying to catch failed blocks by making the output of each task optional (using <code>File?</code>) and then having an <code>if(defined(taskX.out))</code> condition to call the next task. My reasoning is that if something fails (represented by the <code>if(...)</code> within the command block of each task), then the <code>out</code> file will never be created in that task and so won't exist as input for the next task and thus <code>if(defined(taskX.out))</code> will resolve as false and skip the next task (and therefore all the subsequent tasks).</p>
<p>With current testing, the first file in the list satisfies the <code>if(...)</code> conditions of all the tasks and succeeds in giving me the final output file <code>firstfile_1_2_3.txt</code>. However, I am getting a <code>Required file output 'secondfile_1.txt' does not exist'</code> error message because it fails the <code>if(...)</code> condition in <code>task1</code> and so the output file is never created.</p>
<p>From what I've read in the <a href="https://github.com/openwdl/wdl/blob/main/versions/development/SPEC.md#declarations" rel="nofollow noreferrer">WDL v1.1 spec</a>, optional <code>File</code> outputs are allowed and <a href="https://github.com/openwdl/wdl/blob/main/versions/development/SPEC.md#file-directory-and-optional-outputs" rel="nofollow noreferrer">are undefined if the files they point to don't exist</a>. However, after much confused googling, I've realised it's probably because I'm using <a href="https://github.com/broadinstitute/cromwell" rel="nofollow noreferrer">Cromwell</a>, which is stuck with WDL v1.0.</p>
<p>My question is whether this should be working in WDL v1.0 and, if not, is my logic correct and should this (psuedo-)code work with WDL v1.1? Is there a way to achieve what I'm trying to do using pure WDL v1.0?</p>

CC BY-SA 4.0
2022-05-04T06:52:01.603Z
@M__ Thanks - my confusion is based around the different specs. It seems that the features I'm trying to use (optional files) don't seem to be supported in the spec that Cromwell uses.
tiit_helimut
CC BY-SA 4.0
2022-05-04T06:55:54.347Z
@DavyCats Thanks for the comment! I think I've made the mistake to assume that Cromwell incorporates the latest drafts, but it actually uses WDL 1.0 and so these features are not supported. Despite the WDL 1.1 spec being > 1 year old, I can't see any Cromwell implementation based around it yet.
tiit_helimut
CC BY-SA 4.0
2022-03-09T18:59:04.813Z
I stayed clear of WDL because seemed to over complicate things. Can re-write this as a bash script but would need to study the pipeline a bit more.
M__
CC BY-SA 4.0
2022-03-09T19:00:10.357Z
I think the thing to do is contact Broad Institute directly because this appears to be a version issue pertaining the upgrade from 1.0 to 1.1
M__
CC BY-SA 4.0
2022-03-11T11:33:05.107Z
Your WDL code looks more like draft-2 WDL than 1.0 or 1.1 WDL. In WDL 1.0/1.1 there is a separate input section in tasks and workflows, and the version must be specified on the first line. In any case, optional file outputs should be supported in all versions.
DavyCats
<wdl><workflow-management>
Handling failed blocks within scattered WDL workflow
# Handling failed blocks within scattered WDL workflow

(Asked by: username_0 on 2022-03-08)

I'm trying to adapt an existing workflow to make it more robust to failures. Below is a simplified example of the workflow.
workflow main {
   File filelist="list_of_input_files.txt"
   Array[String] files=read_lines(filelist)
   scatter(file in files) {
      call task1 { input: in=file }
      if(defined(task1.out)) { call task2 { input: in=task1.out } }
      if(defined(task2.out)) { call task2 { input: in=task2.out }
   }
}

task task1 {
   File in
   String out1 = basename(in,'.txt') + '_1.txt'
   command { if(...) { ./script1.sh ${in} > ${out1} } }
   output { File? out = out1 }
   runtime { ... }
}

task task2 {
   File in
   String out2 = basename(in,'.txt') + '_2.txt'
   command { if(...) {./script2.sh ${in} > ${out2} } }
   output { File? out = out2 }
   runtime { ... }
}

task task3 {
   File in
   String out3 = basename(in,'.txt') + '_3.txt'
   command { if(...) {./script3.sh ${in} > ${out3} } }
   output { File? out = out3 }
   runtime {...}
}

with my list_of_input_files.txt looking like
firstfile.txt
secondfile.txt
thirdfile.txt
...

Obviously this is oversimplified, but I am trying to catch failed blocks by making the output of each task optional (using File?) and then having an if(defined(taskX.out)) condition to call the next task. My reasoning is that if something fails (represented by the if(...) within the command block of each task), then the out file will never be created in that task and so won't exist as input for the next task and thus if(defined(taskX.out)) will resolve as false and skip the next task (and therefore all the subsequent tasks).
With current testing, the first file in the list satisfies the if(...) conditions of all the tasks and succeeds in giving me the final output file firstfile_1_2_3.txt. However, I am getting a Required file output 'secondfile_1.txt' does not exist' error message because it fails the if(...) condition in task1 and so the output file is never created.
From what I've read in the WDL v1.1 spec, optional File outputs are allowed and are undefined if the files they point to don't exist. However, after much confused googling, I've realised it's probably because I'm using Cromwell, which is stuck with WDL v1.0.
My question is whether this should be working in WDL v1.0 and, if not, is my logic correct and should this (psuedo-)code work with WDL v1.1? Is there a way to achieve what I'm trying to do using pure WDL v1.0?

**Tags:** wdl, workflow-management
> **Comments:**
> **username_1**: I stayed clear of WDL because seemed to over complicate things. Can re-write this as a bash script but would need to study the pipeline a bit more.  
> **username_1**: I think the thing to do is contact Broad Institute directly because this appears to be a version issue pertaining the upgrade from 1.0 to 1.1  
> **username_2**: Your WDL code looks more like draft-2 WDL than 1.0 or 1.1 WDL. In WDL 1.0/1.1 there is a separate input section in tasks and workflows, and the version must be specified on the first line. In any case, optional file outputs should be supported in all versions.  
> **username_0**: @M__ Thanks - my confusion is based around the different specs. It seems that the features I'm trying to use (optional files) don't seem to be supported in the spec that Cromwell uses.  
> **username_0**: @DavyCats Thanks for the comment! I think I've made the mistake to assume that Cromwell incorporates the latest drafts, but it actually uses WDL 1.0 and so these features are not supported. Despite the WDL 1.1 spec being > 1 year old, I can't see any Cromwell implementation based around it yet.  

---

## 1 answers

### Answer by username_2 on 2022-05-05. Score: 1

Optional outputs should work in WDL 1.0. Here is an example that runs without issue through cromwell (version 74).
Note that for the second call in the scatter the input file needs to be converted to from an optional to an non optional value. Since this only gets run if this value is defined, this conversion can safely be done using the select_first function (which returns the first defined value in a given list of optionals).
version 1.0

workflow test {
    input {
        File filelist = "files.txt"
    }
    Array[String] files = read_lines(filelist)

    scatter (file in  files) {
         call blah as b1 {input: in=file}
         if (defined(b1.out)) {
             call blah as b2 {input: in=select_first([b1.out])}
         }
    }
}


task blah {
    input {
        File in
    }
    String outPath = basename(in,'.txt') + '_blah.txt'

    command {
        if $(grep -q "yay" ~{in})
          then
             echo yay > ~{outPath}
        fi
    }

    output {
        File? out = outPath
    }
}

files.txt looks like this:
test1.txt
test2.txt

With test1.txt containing yay and test2.txt containing nay.


---

