<p>You could use the <a href="https://github.com/openwdl/wdl/blob/master/versions/1.0/SPEC.md#string-substring-string-string" rel="nofollow noreferrer"><code>sub</code></a> function to obtain the chromosome from the globbed file names and then use that to retrieve the correct LD matrices from a Map (see <a href="https://github.com/openwdl/wdl/blob/master/versions/1.0/SPEC.md#mapstring-string-read_mapstringfile" rel="nofollow noreferrer"><code>read_map</code></a> or <a href="https://github.com/openwdl/wdl/blob/master/versions/1.0/SPEC.md#mixed-read_jsonstringfile" rel="nofollow noreferrer"><code>read_json</code></a>).</p>

<p>eg.:</p>

<pre><code>Map[String, String] ld_matrices = read_json(file_of_ld_matrices_by_chr)

scatter(gwas_for_chr in split.gwas_by_chr) {
    String chr = sub(sub(gwas_for_chr, prefix, ""), "\.assoc$", "")

    call sbayes.run {
        input:
            GCTB = gctb_executable_path,
            gwas = gwas_for_chr,
            ld_matrix = ld_matrices[chr],
            output_prefix = out
    }
}
</code></pre>

<p>The <code>file_of_ld_matrices_by_chr</code> would have to look like this:</p>

<pre><code>{
  "chr1": "/path/to/file1",
  "chr2": "/path/to/file2"
}
</code></pre>

CC BY-SA 4.0
2020-04-27T15:20:41.120Z
This works perfectly.
Vivek
CC BY-SA 4.0
2020-04-27T14:59:49.550Z
2020-04-27T14:59:49.550Z
DavyCats
<p>I'm trying to implement a polygenic score pipeline in WDL and I'm quite new to this pipeline management.</p>

<p>The first step of the pipeline takes a file of GWAS summary statistics and splits it by chromosome.</p>

<p>The corresponding wdl task:</p>

<pre><code>task split {
    input {
        File gwas
        String output_prefix
    }

    command {
        ./splitGwas -i <span class="math-container">${gwas} -o $</span>{output_prefix}
    }

    output {
        Array [File] gwas_by_chr = glob("${output_prefix}_*.assoc")
    }
}
</code></pre>

<p>The next part of the process is to compute posterior effects of SNPs by chromosome. The inputs to this task are going to be the files split by chromosome in the previous step and an LD matrix for the chromosome.</p>

<p>The corresponding wdl call would be something like this:</p>

<pre><code>Array [File] ld_matrices = read_lines(file_of_ld_matrices_by_chr)

scatter(pair in zip(split.gwas_by_chr, ld_matrices)) {
    call sbayes.run {
        input:
            GCTB = gctb_executable_path,
            gwas = pair.left,
            ld_matrix = pair.right,
            output_prefix = out
    }
}
</code></pre>

<p>Since I'm grabbing the split files using glob in the previous step, I do not particularly know which element in the array corresponds to which chromosome. So I would be making a mistake if I zip the summary stats for one chromosome with the LD matrix corresponding to a different chromosome.</p>

<p>The programmatic way would be to iterate the scatter over chromosome name but then I would lose the ability to specify the dependency between the two tasks.</p>

<p>Is there a better way to do this?</p>

<p>Cross posted on biostars (<a href="https://www.biostars.org/p/434582/" rel="nofollow noreferrer">https://www.biostars.org/p/434582/</a>) so pardon me if that is discouraged.</p>

CC BY-SA 4.0
2020-04-27T11:47:59.223Z
Two questions: 1. Would it be possible for `file_of_ld_matrices_by_chr` to be in JSON format, instead of just a list a list of file paths? 2. What does the name of the file produced by `splitGwas` look like (ie. what's in between the prefix and the `.assoc`?
DavyCats
CC BY-SA 4.0
2020-04-27T14:31:20.837Z
1. Yes, I can put the LD matrices in JSON
2. The file names produced by split GWAS have the names ${prefix}_chromosome.assoc so if the user supplied prefix string is T2D as in type 2 diabetes, the outputs are T2D_1.assoc, T2D_2.assoc ... T2D_22.assoc
Vivek
<wdl>
Matching files by chromosome during wdl scatter gather
# Matching files by chromosome during wdl scatter gather

(Asked by: username_0 on 2020-04-27)

I'm trying to implement a polygenic score pipeline in WDL and I'm quite new to this pipeline management.
The first step of the pipeline takes a file of GWAS summary statistics and splits it by chromosome.
The corresponding wdl task:
task split {
    input {
        File gwas
        String output_prefix
    }

    command {
        ./splitGwas -i ${gwas} -o ${output_prefix}
    }

    output {
        Array [File] gwas_by_chr = glob("${output_prefix}_*.assoc")
    }
}

The next part of the process is to compute posterior effects of SNPs by chromosome. The inputs to this task are going to be the files split by chromosome in the previous step and an LD matrix for the chromosome.
The corresponding wdl call would be something like this:
Array [File] ld_matrices = read_lines(file_of_ld_matrices_by_chr)

scatter(pair in zip(split.gwas_by_chr, ld_matrices)) {
    call sbayes.run {
        input:
            GCTB = gctb_executable_path,
            gwas = pair.left,
            ld_matrix = pair.right,
            output_prefix = out
    }
}

Since I'm grabbing the split files using glob in the previous step, I do not particularly know which element in the array corresponds to which chromosome. So I would be making a mistake if I zip the summary stats for one chromosome with the LD matrix corresponding to a different chromosome.
The programmatic way would be to iterate the scatter over chromosome name but then I would lose the ability to specify the dependency between the two tasks.
Is there a better way to do this?
Cross posted on biostars (https://www.biostars.org/p/434582/) so pardon me if that is discouraged.

**Tags:** wdl
> **Comments:**
> **username_1**: Two questions: 1. Would it be possible for `file_of_ld_matrices_by_chr` to be in JSON format, instead of just a list a list of file paths? 2. What does the name of the file produced by `splitGwas` look like (ie. what's in between the prefix and the `.assoc`?  
> **username_0**: 1. Yes, I can put the LD matrices in JSON 2. The file names produced by split GWAS have the names ${prefix}_chromosome.assoc so if the user supplied prefix string is T2D as in type 2 diabetes, the outputs are T2D_1.assoc, T2D_2.assoc ... T2D_22.assoc  

---

## 1 answers

### Answer by username_1 on 2020-04-27. Score: 1 | Accepted answer

You could use the sub function to obtain the chromosome from the globbed file names and then use that to retrieve the correct LD matrices from a Map (see read_map or read_json).
eg.:
Map[String, String] ld_matrices = read_json(file_of_ld_matrices_by_chr)

scatter(gwas_for_chr in split.gwas_by_chr) {
    String chr = sub(sub(gwas_for_chr, prefix, ""), "\.assoc$", "")

    call sbayes.run {
        input:
            GCTB = gctb_executable_path,
            gwas = gwas_for_chr,
            ld_matrix = ld_matrices[chr],
            output_prefix = out
    }
}

The file_of_ld_matrices_by_chr would have to look like this:
{
  "chr1": "/path/to/file1",
  "chr2": "/path/to/file2"
}

> **Comments:**
> **username_0**: This works perfectly.  

---

