<p>Here's an alternative to using a custom task, albeit still rather unappetizing:</p>

<pre><code>workflow groupby {
  input {
    File? nil
  }

  Array[File] X = ["file1", "file2", "file3", "file4", "file5"]
  Array[String] Y  =  ["A",     "A",     "B",     "A",     "B"]

  scatter (y in ["A", "B"]) {
    scatter (p in zip(X,Y)) {
      File? x_or_nil = if (p.right == y) then p.left else nil
    }
    Pair[String,Array[File]] group = (y, select_all(x_or_nil))
  }

  output {
    Array[Pair[String,Array[File]]] ans = group
  }
}
</code></pre>

<p>(It'll be slightly cleaner with the upcoming <code>None</code> literal instead of dummy <code>nil</code>)</p>

<p>FWIW, <a href="https://github.com/openwdl/wdl/issues/312" rel="nofollow noreferrer">here's a lengthy discussion</a> about the idea of allowing developers to define custom library functions which just run a small task that manipulates data structures (hinting that it needn't necessarily offload to a worker node, with the huge latency that typically entails).</p>

CC BY-SA 4.0
2020-06-17T16:52:00.290Z
When you use `y in ["A", "B"]` you are doing something that assumes you already know the unique contents of Y. This could be achieved using the `keys()` function but that is also only present in the development version and if the development version is to be used the solution with `collect_by_key()` is more minimalist.
Giulio Genovese
CC BY-SA 4.0
2020-06-16T20:40:59.630Z
2020-06-16T20:40:59.630Z
Mike Lin
<p>It turns out the development version has a function <code>collect_by_key</code> that is particularly helpful for scattering around predefined batches:</p>

<pre><code>Array[String] X  =  ["A",     "A",     "B",     "A",     "B"]
Array[File] Y = ["file1", "file2", "file3", "file4", "file5"]
scatter (pair in as_pairs(collect_by_key(zip(X,Y)))) {
  call test {input: X = pair.left, Y = pair.right}
}
</code></pre>

<p>This will call task <code>t</code> first with input:</p>

<pre><code>String X = "A"
Array[File] Y =["file1", "file2", "file4"]
</code></pre>

<p>And then with input:</p>

<pre><code>String X = "B"
Array[File] Y =["file3", "file5"]
</code></pre>

<p>This is a readable solution to this problem allowed by <code>collect_by_key</code>. This function is not very well advertised (I had completely missed it) but kudos to who introduced it in the language. The only drawback is that <code>collect_by_key</code> is not available in version 1.0.</p>

CC BY-SA 4.0
2020-06-17T16:49:03.970Z
2020-06-17T16:49:03.970Z
Giulio Genovese
<p>Suppose I have two arrays Array[File] X and Array[String] Y both of length N, with Y being a sort of R-like factor that defines batch membership for X, for a total of M batches with M &lt; N. A valid example would be:</p>

<pre><code>Array[File] X = ["file1", "file2", "file3", "file4", "file5"]
Array[String] Y  =  ["A",     "A",     "B",     "A",     "B"]
</code></pre>

<p>In which case N=5 and M=2. I want to scatter M tasks so that each task takes only the files corresponding to the same batch. In the previous example, one task would take as input:</p>

<pre><code>Array[File] X =["file1", "file2", "file4"]
String Y = "A"
</code></pre>

<p>And the other task would take as input:</p>

<pre><code>Array[File] X =["file3", "file5"]
String Y = "B"
</code></pre>

<p>Is there an easy way to do this within a WDL workflow in the general case? I understand that there are less elegant ways to do this as its own task but, as this seems like it could be a general purpose scatter/gather problem, an easy solution might be of general interest to the community.</p>

<wdl>
Scatter tasks around predefined batches inside a WDL workflow
# Scatter tasks around predefined batches inside a WDL workflow

(Asked by: username_0 on 2020-06-16)

Suppose I have two arrays Array[File] X and Array[String] Y both of length N, with Y being a sort of R-like factor that defines batch membership for X, for a total of M batches with M < N. A valid example would be:
Array[File] X = ["file1", "file2", "file3", "file4", "file5"]
Array[String] Y  =  ["A",     "A",     "B",     "A",     "B"]

In which case N=5 and M=2. I want to scatter M tasks so that each task takes only the files corresponding to the same batch. In the previous example, one task would take as input:
Array[File] X =["file1", "file2", "file4"]
String Y = "A"

And the other task would take as input:
Array[File] X =["file3", "file5"]
String Y = "B"

Is there an easy way to do this within a WDL workflow in the general case? I understand that there are less elegant ways to do this as its own task but, as this seems like it could be a general purpose scatter/gather problem, an easy solution might be of general interest to the community.

**Tags:** wdl

---

## 2 answers

### Answer by username_1 on 2020-06-16. Score: 1

Here's an alternative to using a custom task, albeit still rather unappetizing:
workflow groupby {
  input {
    File? nil
  }

  Array[File] X = ["file1", "file2", "file3", "file4", "file5"]
  Array[String] Y  =  ["A",     "A",     "B",     "A",     "B"]

  scatter (y in ["A", "B"]) {
    scatter (p in zip(X,Y)) {
      File? x_or_nil = if (p.right == y) then p.left else nil
    }
    Pair[String,Array[File]] group = (y, select_all(x_or_nil))
  }

  output {
    Array[Pair[String,Array[File]]] ans = group
  }
}

(It'll be slightly cleaner with the upcoming None literal instead of dummy nil)
FWIW, here's a lengthy discussion about the idea of allowing developers to define custom library functions which just run a small task that manipulates data structures (hinting that it needn't necessarily offload to a worker node, with the huge latency that typically entails).

> **Comments:**
> **username_0**: When you use `y in ["A", "B"]` you are doing something that assumes you already know the unique contents of Y. This could be achieved using the `keys()` function but that is also only present in the development version and if the development version is to be used the solution with `collect_by_key()` is more minimalist.  

---

### Answer by username_0 on 2020-06-17. Score: 0

It turns out the development version has a function collect_by_key that is particularly helpful for scattering around predefined batches:
Array[String] X  =  ["A",     "A",     "B",     "A",     "B"]
Array[File] Y = ["file1", "file2", "file3", "file4", "file5"]
scatter (pair in as_pairs(collect_by_key(zip(X,Y)))) {
  call test {input: X = pair.left, Y = pair.right}
}

This will call task t first with input:
String X = "A"
Array[File] Y =["file1", "file2", "file4"]

And then with input:
String X = "B"
Array[File] Y =["file3", "file5"]

This is a readable solution to this problem allowed by collect_by_key. This function is not very well advertised (I had completely missed it) but kudos to who introduced it in the language. The only drawback is that collect_by_key is not available in version 1.0.


---

