<p>MAGIC assumes input data has been both library-size normalized, and either log or sqrt transformed prior to imputation (see also: <a href="http://nbviewer.jupyter.org/github/KrishnaswamyLab/magic/blob/master/python/tutorial_notebooks/bonemarrow_tutorial.ipynb" rel="nofollow noreferrer">MAGIC tutorial</a>). Additionally, any graph-based methods (MAGIC, PHATE, t-SNE, UMAP, spectral clustering, Louvain, etc etc) will give flawed results if your data contains a batch effect, since the neighbourhood graph would reflect that structure of your batch effect, and worse, imputation would further reinforce this batch effect.</p>

<p>Thus I would recommend the following pipeline:</p>

<ul>
<li>Library-size normalization</li>
<li>Square root (or log) transform</li>
<li>Batch effect removal</li>
<li>Imputation</li>
</ul>

<p>Regarding your <strong>update</strong>, the reason you don't see the batch effect in the raw counts data is simply that the batch effect is not visible in the most highly expressed genes. Prior to transformation, the principal source of variation in your data is simply the expression of the most highly expressed genes - this is essentially masking the source of the batch effect, not removing it. I recommend never working with raw molecule counts in scRNAseq, as the raw counts data hides much of the heterogenity in your dataset, which is precisely what you are looking in doing single-cell RNA-seq.</p>

CC BY-SA 4.0
2018-10-21T18:35:04.123Z
2018-10-21T18:35:04.123Z
Scott Gigante
<p>I am preprocessing scRNA-seq data. What is the best practice in use to run both ComBat for batch effects removal, data imputation (to mitigate dropout) and library size normalization?</p>

<p>I thought that library size should be run first, since it is per-cell normalization, then ComBat batch effects removal. On the original paper - <a href="https://academic.oup.com/biostatistics/article/8/1/118/252073" rel="nofollow noreferrer">Johnson et al. (2007)</a> - it is stated that:</p>

<blockquote>
  <p>We assume that the data have been normalized and expression values have been estimated for all genes and samples.</p>
</blockquote>

<p>However, I want to apply it to scRNA-seq data. Does this statement still hold? Additionally, I plan to apply imputation (e.g. with <a href="https://www.biorxiv.org/content/early/2017/02/25/111591" rel="nofollow noreferrer">MAGIC</a>) in the end. Is there any problem you can spot?</p>

<h3>Update</h3>

<p>I attach the PCA regarding an example Mus Musculus dataset in which different colors represent different mice. It seems clear to me that the first two principal components are affected by batches (mouse id).</p>

<p><a href="https://i.sstatic.net/TeZGZ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/TeZGZ.png" alt="pca"></a></p>

<h3>Update 2</h3>

<p>I rerun the PCA on raw counts data (the first PCA was on log-transformed data) and I obtain a different description of the dataset, in which batch effects seem not to be prevalent.</p>

<p><a href="https://i.sstatic.net/OfSUZ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/OfSUZ.png" alt="pca_raw"></a></p>

CC BY-SA 3.0
2018-01-04T00:00:11.140Z
From what I can tell, MAGIC should be run on raw data, so that would be the first step.
burger
CC BY-SA 3.0
2018-01-04T16:32:08.340Z
@burger MAGIC normalizes data before imputation, so it should be run at least after library size normalization. My concern is that using MAGIC before ComBat will amplify batch effects. Reading the paper I could not find any reference to batch effect removal.
gc5
CC BY-SA 3.0
2018-01-10T09:39:27.633Z
The advice I got was that the best would be to adjust for batch effect instead of removing them. Did you tried adjusting for your batch effects? How big is your batch effect ? (Do the PCA or MDS or dendograms show a clear distinction by your batch effect (or several batchs) ?)
llrs
CC BY-SA 3.0
2018-01-10T14:34:03.857Z
@Llopis yes, actually for batch effects removal I meant adjusting for batch effect with ComBat, is it what you meant?
gc5
CC BY-SA 3.0
2018-01-10T14:57:02.350Z
No, comBat doesn't adjust for batch effect it "removes" it (Despite the first line of the help page). From the (same) help page: "Users are returned an expression matrix that has been corrected for batch effects"; it modifies the data to "adjust" for it, instead of adding/calculating a factor to be taken into account for later steps. The later can be done in limma, DESeq2 and other packages but it is not the same adjusting than removing.
llrs
CC BY-SA 3.0
2018-01-10T15:03:46.803Z
@Llopis ok thanks, I didn't know this distinction. However, my PCA shows a clear distinction between batches. I'll update the question with the figure. Can you please elaborate more on calculating a factor to be taken into account for later steps? Do you mean to extract the principal component correlated with batch and later do something with it?
gc5
CC BY-SA 3.0
2018-01-10T15:06:15.827Z
Well, what I do is include the known batch effects on the linear models. That could be via the component of the PCA or the categories you know of your batches. Could you expand into what are your batches ? What are 3_8, 3_38 ... (I assume M is male and F is Female).?
llrs
CC BY-SA 3.0
2018-01-10T15:20:17.350Z
@Llopis unfortunately I have no metadata information regarding the first part of the mouse id. They only provide sex (M/F) which is as you assumed.
gc5
CC BY-SA 3.0
2018-01-10T15:24:15.597Z
Your PC2 separates cells of the mice but is only 0.6% of the variation, so I would say there isn't batch effect. The first dimension is quite high, but I don't know if this is normal in scRNA-seq. I wouldn't adjust nor remove batch effect here if this would be RNA-seq. But I have never analysed scRNA
llrs
CC BY-SA 3.0
2018-01-10T15:48:33.910Z
Let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/71464/discussion-between-gc5-and-llopis).
gc5
CC BY-SA 3.0
2018-01-25T15:01:21.040Z
in my experience, the absolute first thing you need to do is normalize for library size. I suspect that if you color your cells according to size you will notice a clear correlation with PC1.
galicae
<normalization><scrnaseq><imputation><combat><batch-effects>
Order of batch effects removal, data imputation and library size normalization in scRNA-seq data
# Order of batch effects removal, data imputation and library size normalization in scRNA-seq data

(Asked by: username_0 on 2018-01-03)

I am preprocessing scRNA-seq data. What is the best practice in use to run both ComBat for batch effects removal, data imputation (to mitigate dropout) and library size normalization?

I thought that library size should be run first, since it is per-cell normalization, then ComBat batch effects removal. On the original paper - [Johnson et al. (2007)](https://academic.oup.com/biostatistics/article/8/1/118/252073) - it is stated that:

> We assume that the data have been normalized and expression values have been estimated for all genes and samples.

However, I want to apply it to scRNA-seq data. Does this statement still hold? Additionally, I plan to apply imputation (e.g. with [MAGIC](https://www.biorxiv.org/content/early/2017/02/25/111591)) in the end. Is there any problem you can spot?

### Update

I attach the PCA regarding an example Mus Musculus dataset in which different colors represent different mice. It seems clear to me that the first two principal components are affected by batches (mouse id).

[![pca](https://i.sstatic.net/TeZGZ.png)](https://i.sstatic.net/TeZGZ.png)

### Update 2

I rerun the PCA on raw counts data (the first PCA was on log-transformed data) and I obtain a different description of the dataset, in which batch effects seem not to be prevalent.

[![pca_raw](https://i.sstatic.net/OfSUZ.png)](https://i.sstatic.net/OfSUZ.png)

**Tags:** normalization, scrnaseq, imputation, combat, batch-effects
> **Comments:**
> **username_1**: From what I can tell, MAGIC should be run on raw data, so that would be the first step.  
> **username_0**: @burger MAGIC normalizes data before imputation, so it should be run at least after library size normalization. My concern is that using MAGIC before ComBat will amplify batch effects. Reading the paper I could not find any reference to batch effect removal.  
> **username_2**: The advice I got was that the best would be to adjust for batch effect instead of removing them. Did you tried adjusting for your batch effects? How big is your batch effect ? (Do the PCA or MDS or dendograms show a clear distinction by your batch effect (or several batchs) ?)  
> **username_0**: @Llopis yes, actually for batch effects removal I meant adjusting for batch effect with ComBat, is it what you meant?  
> **username_2**: No, comBat doesn't adjust for batch effect it "removes" it (Despite the first line of the help page). From the (same) help page: "Users are returned an expression matrix that has been corrected for batch effects"; it modifies the data to "adjust" for it, instead of adding/calculating a factor to be taken into account for later steps. The later can be done in limma, DESeq2 and other packages but it is not the same adjusting than removing.  
> **username_0**: @Llopis ok thanks, I didn't know this distinction. However, my PCA shows a clear distinction between batches. I'll update the question with the figure. Can you please elaborate more on calculating a factor to be taken into account for later steps? Do you mean to extract the principal component correlated with batch and later do something with it?  
> **username_2**: Well, what I do is include the known batch effects on the linear models. That could be via the component of the PCA or the categories you know of your batches. Could you expand into what are your batches ? What are 3_8, 3_38 ... (I assume M is male and F is Female).?  
> **username_0**: @Llopis unfortunately I have no metadata information regarding the first part of the mouse id. They only provide sex (M/F) which is as you assumed.  
> **username_2**: Your PC2 separates cells of the mice but is only 0.6% of the variation, so I would say there isn't batch effect. The first dimension is quite high, but I don't know if this is normal in scRNA-seq. I wouldn't adjust nor remove batch effect here if this would be RNA-seq. But I have never analysed scRNA  
> **username_0**: Let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/71464/discussion-between-gc5-and-llopis).  
> **username_3**: in my experience, the absolute first thing you need to do is normalize for library size. I suspect that if you color your cells according to size you will notice a clear correlation with PC1.  

---

## 1 answers

### Answer by username_4 on 2018-10-21. Score: 3

MAGIC assumes input data has been both library-size normalized, and either log or sqrt transformed prior to imputation (see also: [MAGIC tutorial](http://nbviewer.jupyter.org/github/KrishnaswamyLab/magic/blob/master/python/tutorial_notebooks/bonemarrow_tutorial.ipynb)). Additionally, any graph-based methods (MAGIC, PHATE, t-SNE, UMAP, spectral clustering, Louvain, etc etc) will give flawed results if your data contains a batch effect, since the neighbourhood graph would reflect that structure of your batch effect, and worse, imputation would further reinforce this batch effect.

Thus I would recommend the following pipeline:

* Library-size normalization
* Square root (or log) transform
* Batch effect removal
* Imputation

Regarding your **update**, the reason you don't see the batch effect in the raw counts data is simply that the batch effect is not visible in the most highly expressed genes. Prior to transformation, the principal source of variation in your data is simply the expression of the most highly expressed genes - this is essentially masking the source of the batch effect, not removing it. I recommend never working with raw molecule counts in scRNAseq, as the raw counts data hides much of the heterogenity in your dataset, which is precisely what you are looking in doing single-cell RNA-seq.


---

