<p>Negative values are to be expected as you have log-transformed the data. Any data value below 2 will have a negative log2.</p>

<p>There is a difference between normalising for analysis and truncating for plotting. The RMA normalisation or similar is intended for differential expression analysis. I would recommend using this normalisation first.</p>

<p>When plotting, the normalised data may still contain outliers which skew the plotting parameters (such as the colour scale). Due to the large number of genes, there may be some far beyond 3 standard deviations and plotting these at their true value will make it hard to discern trends in the middle. Truncating the data for a heatmap is common, typically at |z| > 3 but quantiles should work much the same. This is a matter of visualisation, not clustering analysis. It would not be needed unless you were plotting the data.</p>

CC BY-SA 3.0
2017-12-05T09:08:21.453Z
Hello Tom, Yes I want to plot these data. You mean I should use the normalisation which is mentioned above in my question?
beginner
CC BY-SA 3.0
2017-12-12T07:15:03.427Z
You should normalise the data first as you wish to analyse it further. The visualisations should as closely matched the data you've used, to adjust for batch effects, etc. If necessary to plot the values on a sensible range further standardisation (such as the z-transform) may be done. While there are best practices, expression analysis is not an exact science and involves a lot of decisions, depending on the variation you want to focus on, for what you can show from the data.
Tom Kelly ケリー・トム
CC BY-SA 3.0
2017-11-26T03:26:46.203Z
2017-11-26T03:26:46.203Z
Tom Kelly ケリー・トム
<p>You see negative values with your function because you're setting the average of each row to 0 and its standard deviation to 1.</p>

<p>In general, I would trust a standard normalization method (rma in this case) more than some random "truncate and then scale the rows" method. Your method isn't even doing any between-array normalization, which is the benefit of rma.</p>

CC BY-SA 3.0
2017-11-06T20:51:42.013Z
Thankyou for the reply. So, I will use the rma normalisation method.                                                 ty <- backgroundCorrect(data); normdata <- normalize(ty); And then using the normdata I can generate a clustering heat map. Am I right?
beginner
CC BY-SA 3.0
2017-11-07T10:40:36.833Z
Could you please tell me whether what I mentioned in my previous comment is right or not?
beginner
CC BY-SA 3.0
2017-11-07T10:47:57.103Z
That or `rma()` directly.
Devon Ryan
CC BY-SA 3.0
2017-11-07T12:22:59.000Z
Ok. I did the normalisation like I mentioned in my previous comment. I also generated a clustering heat map. Could please have a look and tell me whether this is fine or not for clustering? https://imgur.com/3vZsO4Y
beginner
CC BY-SA 3.0
2017-11-07T12:24:27.660Z
It looks like a clustered heatmap, whether it's "fine" is something that only you can answer.
Devon Ryan
CC BY-SA 3.0
2017-11-07T12:27:33.553Z
Yes, I know what I mean is whether there is anything needs to be added for clustering heat map. Thank you !!
beginner
CC BY-SA 3.0
2017-11-07T12:29:24.950Z
There isn't anything else absolutely required. It might be nice to not split by `Type`, since I imagine someone would like to see if the samples cluster according to it.
Devon Ryan
CC BY-SA 3.0
2017-11-07T12:37:25.987Z
Ok. But when I don't use split it looks like this. https://imgur.com/a/MJotE
beginner
CC BY-SA 3.0
2017-11-07T12:38:33.780Z
That's telling you something that's rather important to know.
Devon Ryan
CC BY-SA 3.0
2017-11-07T12:42:28.167Z
This means samples are not clustered. Am I right?
beginner
CC BY-SA 3.0
2017-11-07T12:43:21.060Z
They don't cluster by `Type`, at least.
Devon Ryan
CC BY-SA 3.0
2017-11-06T20:35:15.903Z
2017-11-06T20:35:15.903Z
Devon Ryan
<p>I wanted to generate a clustering heat map for the microarray data. This is the first time I'm working on Microarray data. I read some tutorials but have few doubts.</p>

<p>I'm using microarray (Affymetrix SNP 6.0 data) gene expression data. For example the data looks like following:</p>

<pre><code>ProbeID S1  S2  S3  S4  S5  S6  S7
10008   131.4311    369.4926    222.0441    687.4181    176.8892    258.1233    316.5573
10010   78.73022    83.97501    81.56039    86.11443    78.09758    81.88231    84.17101
10014   90.02816    95.07267    101.1761    93.35585    81.96468    94.3553 93.89527
10017   79.86837    81.63064    88.19524    79.47265    76.3437 101.6351    93.71674
10019   99.03493    109.1835    104.97  102.7423    108.3677    98.93459    101.4052
10020   79.58075    84.28915    90.53562    74.47786    75.96112    96.39649    95.8828
10021   121.5373    149.9351    146.5956    122.8523    110.5759    132.4268    130.4409
10025   616.5994    1326.735    1358.187    2315.851    1068.745    3229.759    4435.021
10035   70.44073    69.56772    68.25446    68.35857    70.86771    74.3843 67.93569
</code></pre>

<p>I created expression matrix and did log2 conversion. Now the data looks like following.</p>

<pre><code>            S1  S2  S3  S4  S5  S6  S7
Gene1   6.429276339 6.338451158 6.333760753 6.419191996 6.503471181 6.329103499 6.211373601
Gene2   6.379471993 6.296018518 6.237316465 6.2696332   6.329489132 6.359770303 6.240070336
Gene3   12.84498365 12.00265682 13.92741965 12.553162   13.39001307 13.8933423  12.58695704
Gene4   8.661860382 6.723004202 6.300975176 7.661012019 7.905219709 6.957578023 6.70945883
Gene5   6.945382967 6.814979733 6.575916303 6.63460198  6.627380524 6.733926424 6.280618235
Gene6   9.280222581 8.81560969  9.683073561 9.480038673 8.801438707 9.190943942 7.743705471
Gene7   6.593564871 6.63502488  6.389962535 6.511360029 6.694572404 6.6492763   6.527199544
Gene8   6.431615372 6.309515078 6.248153876 6.329288965 6.46768078  6.355268547 6.384284754
Gene9   7.513349406 7.234654595 7.490935892 6.801368215 7.323811386 7.018733196 7.055932044
</code></pre>

<p>Before clustering for normalisation:</p>

<p>I applied a function for normalisation </p>

<pre><code>data = t(apply(data, 1, function(x) {
  q10 = quantile(x, 0.1)
  q90 = quantile(x, 0.9)
  x[x &lt; q10] = q10
  x[x &gt; q90] = q90
  scale(x)
}))
</code></pre>

<p>When I apply the above function I see there are values which are positive and also negative. </p>

<p>And when I used "rma" for background correction and "normalise" function from "oligo" package I don't see any negative values.</p>

<p>I don't understand. </p>

<p>Which one should I use to check the clustering among samples? Any help is appreciated.</p>

<r><bioconductor><normalization><microarray>
Normalizing microarray data for clustering heat map
# Normalizing microarray data for clustering heat map

(Asked by: username_0 on 2017-11-06)

I wanted to generate a clustering heat map for the microarray data. This is the first time I'm working on Microarray data. I read some tutorials but have few doubts.
I'm using microarray (Affymetrix SNP 6.0 data) gene expression data. For example the data looks like following:
ProbeID S1  S2  S3  S4  S5  S6  S7
10008   131.4311    369.4926    222.0441    687.4181    176.8892    258.1233    316.5573
10010   78.73022    83.97501    81.56039    86.11443    78.09758    81.88231    84.17101
10014   90.02816    95.07267    101.1761    93.35585    81.96468    94.3553 93.89527
10017   79.86837    81.63064    88.19524    79.47265    76.3437 101.6351    93.71674
10019   99.03493    109.1835    104.97  102.7423    108.3677    98.93459    101.4052
10020   79.58075    84.28915    90.53562    74.47786    75.96112    96.39649    95.8828
10021   121.5373    149.9351    146.5956    122.8523    110.5759    132.4268    130.4409
10025   616.5994    1326.735    1358.187    2315.851    1068.745    3229.759    4435.021
10035   70.44073    69.56772    68.25446    68.35857    70.86771    74.3843 67.93569

I created expression matrix and did log2 conversion. Now the data looks like following.
            S1  S2  S3  S4  S5  S6  S7
Gene1   6.429276339 6.338451158 6.333760753 6.419191996 6.503471181 6.329103499 6.211373601
Gene2   6.379471993 6.296018518 6.237316465 6.2696332   6.329489132 6.359770303 6.240070336
Gene3   12.84498365 12.00265682 13.92741965 12.553162   13.39001307 13.8933423  12.58695704
Gene4   8.661860382 6.723004202 6.300975176 7.661012019 7.905219709 6.957578023 6.70945883
Gene5   6.945382967 6.814979733 6.575916303 6.63460198  6.627380524 6.733926424 6.280618235
Gene6   9.280222581 8.81560969  9.683073561 9.480038673 8.801438707 9.190943942 7.743705471
Gene7   6.593564871 6.63502488  6.389962535 6.511360029 6.694572404 6.6492763   6.527199544
Gene8   6.431615372 6.309515078 6.248153876 6.329288965 6.46768078  6.355268547 6.384284754
Gene9   7.513349406 7.234654595 7.490935892 6.801368215 7.323811386 7.018733196 7.055932044

Before clustering for normalisation:
I applied a function for normalisation 
data = t(apply(data, 1, function(x) {
  q10 = quantile(x, 0.1)
  q90 = quantile(x, 0.9)
  x[x < q10] = q10
  x[x > q90] = q90
  scale(x)
}))

When I apply the above function I see there are values which are positive and also negative. 
And when I used "rma" for background correction and "normalise" function from "oligo" package I don't see any negative values.
I don't understand. 
Which one should I use to check the clustering among samples? Any help is appreciated.

**Tags:** r, bioconductor, normalization, microarray

---

## 2 answers

### Answer by username_2 on 2017-11-06. Score: 5 | Accepted answer

You see negative values with your function because you're setting the average of each row to 0 and its standard deviation to 1.

In general, I would trust a standard normalization method (rma in this case) more than some random "truncate and then scale the rows" method. Your method isn't even doing any between-array normalization, which is the benefit of rma.

> **Comments:**
> **username_0**: Thankyou for the reply. So, I will use the rma normalisation method.                                                 ty <- backgroundCorrect(data); normdata <- normalize(ty); And then using the normdata I can generate a clustering heat map. Am I right?  
> **username_0**: Could you please tell me whether what I mentioned in my previous comment is right or not?  
> **username_2**: That or `rma()` directly.  
> **username_0**: Ok. I did the normalisation like I mentioned in my previous comment. I also generated a clustering heat map. Could please have a look and tell me whether this is fine or not for clustering? https://imgur.com/3vZsO4Y  
> **username_2**: It looks like a clustered heatmap, whether it's "fine" is something that only you can answer.  
> **username_0**: Yes, I know what I mean is whether there is anything needs to be added for clustering heat map. Thank you !!  
> **username_2**: There isn't anything else absolutely required. It might be nice to not split by `Type`, since I imagine someone would like to see if the samples cluster according to it.  
> **username_0**: Ok. But when I don't use split it looks like this. https://imgur.com/a/MJotE  
> **username_2**: That's telling you something that's rather important to know.  
> **username_0**: This means samples are not clustered. Am I right?  
> **username_2**: They don't cluster by `Type`, at least.  

---

### Answer by username_1 on 2017-11-26. Score: 1

Negative values are to be expected as you have log-transformed the data. Any data value below 2 will have a negative log2.

There is a difference between normalising for analysis and truncating for plotting. The RMA normalisation or similar is intended for differential expression analysis. I would recommend using this normalisation first.

When plotting, the normalised data may still contain outliers which skew the plotting parameters (such as the colour scale). Due to the large number of genes, there may be some far beyond 3 standard deviations and plotting these at their true value will make it hard to discern trends in the middle. Truncating the data for a heatmap is common, typically at |z| > 3 but quantiles should work much the same. This is a matter of visualisation, not clustering analysis. It would not be needed unless you were plotting the data.

> **Comments:**
> **username_0**: Hello Tom, Yes I want to plot these data. You mean I should use the normalisation which is mentioned above in my question?  
> **username_1**: You should normalise the data first as you wish to analyse it further. The visualisations should as closely matched the data you've used, to adjust for batch effects, etc. If necessary to plot the values on a sensible range further standardisation (such as the z-transform) may be done. While there are best practices, expression analysis is not an exact science and involves a lot of decisions, depending on the variation you want to focus on, for what you can show from the data.  

---

