<blockquote>
  <p>Why use images of reads when you already have the reads data?</p>
</blockquote>

<p>Neural networks take one or multiple <a href="https://docs.scipy.org/doc/numpy-1.13.0/reference/arrays.ndarray.html" rel="nofollow noreferrer">N-dimension arrays</a> (n-d arrays) as input; they can't take arbitrary forms of data. To feed reads and their alignment into the network, you have to encode them as n-d arrays. If you ignore "colors" for now, a straightforward choice is a (windowLength, readDepth) 2D-array as you see in an alignment browser – it is an image. The "colors" of this image are often called "channels" or "features" in deep learning (DL). They represent base types and base quality in deepVariant. In the end, you get a (windowLength, readDepth, nFeatures) 3D array as input. This is the most natural encoding to DL researchers.</p>

<blockquote>
  <p>Why chose CNN for a variant caller</p>
</blockquote>

<p>That is because CNN has been repeatedly proved to be the best architecture for such (height, width, nChannels) 3D arrays as input. Read alignments are exactly that.</p>

<hr>

<p>On a historical note, deepVariant was first built on top of google's now-deprecated <a href="http://www.techtimes.com/articles/104696/20151109/google-announces-machine-learning-system-tensorflow-now-open-sourced.htm" rel="nofollow noreferrer">DisBelief framework</a> that only takes RGB images as input. They had to squeeze all types of information into three color channels. Now with TensorFlow, an image can have an arbitrary number of color channels. This simplifies implementation and improves accuracy. In addition to this image, deepVariant also takes other information as input.</p>

CC BY-SA 3.0
2018-04-19T22:47:29.980Z
2018-04-19T22:47:29.980Z
user172818
<p>It's more "images of reads" rather than actual images of reads. In reality, they're feeding in a pileup, which is just a matrix of letters or numbers (you can visualize this as an image, of course). This is more apparent if you look at <a href="https://github.com/google/deepvariant/blob/r0.6/docs/visualizing_examples.ipynb" rel="nofollow noreferrer">one of their example python notebooks</a>, CNNs are popular in the image processing field, but at the end of the day images are just large matrices of values so any time you can convert your input into that sort of format then a CNN can be useful (provided you have sufficient training data and some GPUs to throw at things).</p>

CC BY-SA 3.0
2018-04-30T17:05:15.850Z
I find the answer confusing: are you saying that the CNN is *not* fed images? Because the way google describes DeepVariant, they are indeed creating pileup image data and putting that through their CNN.
Konrad Rudolph
CC BY-SA 3.0
2018-04-30T18:05:53.007Z
It's not an image, it's an "image". The DeepVariant are a bit ambiguous on that point, though at the end of the day it's a multidimensional matrix of values (the relation of that to an image being obvious).
Devon Ryan
CC BY-SA 3.0
2018-04-19T14:37:20.753Z
2018-04-19T14:37:20.753Z
Devon Ryan
<p>Google released their variant caller <a href="https://github.com/google/deepvariant" rel="nofollow noreferrer">DeepVariant</a> which won the highest SNP performance award in the <a href="https://precision.fda.gov/challenges/truth/results" rel="nofollow noreferrer">Precision FDA Truth challenge</a> (99.999% accuracy).</p>

<p>From the linked github repo, we see that DeepVariant is a CNN, we provide images of the reads as an input and train the neural network with a stochastic gradient descent algorithm. </p>

<p>To me this is unintuitive. Why use images of reads when you already have the reads data?</p>

CC BY-SA 3.0
2018-04-19T14:31:02.840Z
https://en.wikipedia.org/wiki/Law_of_the_instrument
heathobrien
CC BY-SA 3.0
2018-04-19T14:38:33.127Z
Where did you see they need images? The [documentation](https://github.com/google/deepvariant/blob/r0.6/docs/deepvariant-details.md) says the input is a bam file, just like any other caller.
terdon
CC BY-SA 3.0
2018-04-19T14:50:14.860Z
@terdon Images (of pileups, of course) are mentioned [in the preprint](https://www.biorxiv.org/content/early/2018/03/20/092890). I think they're also using the Inception CNN, which is heavily used in image processing. I think many people forget that images are just matrices.
Devon Ryan
CC BY-SA 3.0
2018-04-19T14:50:31.887Z
@terdon It seems that the first version was on images: `We have made a number of improvements to the methodology as well. The biggest change was to move away from RGB-encoded (3-channel) pileup images and instead represent the aligned read data using a multi-channel tensor data layout.` and now they use something else. Thank you!
Claudiu Creanga
CC BY-SA 3.0
2018-04-21T20:14:13.997Z
As @DevonRyan has already said, these are not images per se. From a machine's point of view an image is just a bunch of matrices (one per colour channel). I'd say this is just an example of bad/confusing use of terminology.
Eli Korvigo
<variant-calling>
Why chose CNN for a variant caller
# Why chose CNN for a variant caller

(Asked by: username_0 on 2018-04-19)

Google released their variant caller [DeepVariant](https://github.com/google/deepvariant) which won the highest SNP performance award in the [Precision FDA Truth challenge](https://precision.fda.gov/challenges/truth/results) (99.999% accuracy).

From the linked github repo, we see that DeepVariant is a CNN, we provide images of the reads as an input and train the neural network with a stochastic gradient descent algorithm.

To me this is unintuitive. Why use images of reads when you already have the reads data?

**Tags:** variant-calling
> **Comments:**
> **username_1**: https://en.wikipedia.org/wiki/Law_of_the_instrument  
> **username_2**: Where did you see they need images? The [documentation](https://github.com/google/deepvariant/blob/r0.6/docs/deepvariant-details.md) says the input is a bam file, just like any other caller.  
> **username_3**: @terdon Images (of pileups, of course) are mentioned [in the preprint](https://www.biorxiv.org/content/early/2018/03/20/092890). I think they're also using the Inception CNN, which is heavily used in image processing. I think many people forget that images are just matrices.  
> **username_0**: @terdon It seems that the first version was on images: `We have made a number of improvements to the methodology as well. The biggest change was to move away from RGB-encoded (3-channel) pileup images and instead represent the aligned read data using a multi-channel tensor data layout.` and now they use something else. Thank you!  
> **username_4**: As @DevonRyan has already said, these are not images per se. From a machine's point of view an image is just a bunch of matrices (one per colour channel). I'd say this is just an example of bad/confusing use of terminology.  

---

## 2 answers

### Answer by username_3 on 2018-04-19. Score: 4 | Accepted answer

It's more "images of reads" rather than actual images of reads. In reality, they're feeding in a pileup, which is just a matrix of letters or numbers (you can visualize this as an image, of course). This is more apparent if you look at [one of their example python notebooks](https://github.com/google/deepvariant/blob/r0.6/docs/visualizing_examples.ipynb), CNNs are popular in the image processing field, but at the end of the day images are just large matrices of values so any time you can convert your input into that sort of format then a CNN can be useful (provided you have sufficient training data and some GPUs to throw at things).

> **Comments:**
> **username_6**: I find the answer confusing: are you saying that the CNN is *not* fed images? Because the way google describes DeepVariant, they are indeed creating pileup image data and putting that through their CNN.  
> **username_3**: It's not an image, it's an "image". The DeepVariant are a bit ambiguous on that point, though at the end of the day it's a multidimensional matrix of values (the relation of that to an image being obvious).  

---

### Answer by username_5 on 2018-04-19. Score: 3

> Why use images of reads when you already have the reads data?

Neural networks take one or multiple [N-dimension arrays](https://docs.scipy.org/doc/numpy-1.13.0/reference/arrays.ndarray.html) (n-d arrays) as input; they can't take arbitrary forms of data. To feed reads and their alignment into the network, you have to encode them as n-d arrays. If you ignore "colors" for now, a straightforward choice is a (windowLength, readDepth) 2D-array as you see in an alignment browser – it is an image. The "colors" of this image are often called "channels" or "features" in deep learning (DL). They represent base types and base quality in deepVariant. In the end, you get a (windowLength, readDepth, nFeatures) 3D array as input. This is the most natural encoding to DL researchers.

> Why chose CNN for a variant caller

That is because CNN has been repeatedly proved to be the best architecture for such (height, width, nChannels) 3D arrays as input. Read alignments are exactly that.

---

On a historical note, deepVariant was first built on top of google's now-deprecated [DisBelief framework](http://www.techtimes.com/articles/104696/20151109/google-announces-machine-learning-system-tensorflow-now-open-sourced.htm) that only takes RGB images as input. They had to squeeze all types of information into three color channels. Now with TensorFlow, an image can have an arbitrary number of color channels. This simplifies implementation and improves accuracy. In addition to this image, deepVariant also takes other information as input.


---

