<p>Okay I see. Basically there's a threshold for removing low intensity signal that can't be distinguished from background. That threshold seems tight at 1% of the mean of the normal distributio. Only the signal that is highly significantly above background signal is retained, i.e. subject to further analysis.</p>
<p>The way it works is that for a given CpG site  there is a) methylated allele probe and b) unmethylated allele probe. The beta-value is a representation of those two intensities. What they want to do is filter out the background signal to avoid false positives.</p>
<p>Here's the manual <a href="https://bioconductor.org/packages/devel/bioc/vignettes/minfi/inst/doc/minfi.html#5_Quality_control" rel="nofollow noreferrer">https://bioconductor.org/packages/devel/bioc/vignettes/minfi/inst/doc/minfi.html#5_Quality_control</a></p>
<p>Here's a basic code (I stress the word <em>basic</em>) <strong>HOWEVER</strong> see the note on Noob in minfi below:</p>
<pre><code>library(minfi)

myStuff &lt;- read.metharray.exp(&quot;/Users/username/Desktop/myfile&quot;)
rgSet &lt;- preprocessQuantile(myStuff)

M &lt;- getM(rgSet)
U &lt;- getU(rgSet)

beta &lt;- M / (M + U)

filterProbesByZTest &lt;- function(M, U, p.threshold = 0.01) {
  n &lt;- ncol(M)
  filteredProbes &lt;- rep(TRUE, ncol(M)) 
  
  for (i in 1:ncol(M)) {
    probeM &lt;- M[, i]
    probeU &lt;- U[, i]
    
    meanM &lt;- mean(probeM, na.rm = TRUE)
    meanU &lt;- mean(probeU, na.rm = TRUE)
    sdM &lt;- sd(probeM, na.rm = TRUE)
    sdU &lt;- sd(probeU, na.rm = TRUE)
    
    z_stat &lt;- (meanM - meanU) / sqrt((sdM^2 / length(probeM)) + (sdU^2 / length(probeU)))
    p_value &lt;- 2 * pnorm(-abs(z_stat))
    
    if (p_value &gt; p.threshold) {
      filteredProbes[i] &lt;- FALSE
    }
  }
  
  return(filteredProbes)
}

filteredProbes &lt;- filterProbesByZTest(M, U, p.threshold = 0.01)
betaFiltered &lt;- beta[, filteredProbes]
</code></pre>
<hr />
<p>Note apart from being <em>basic</em>. the code isn't an exact replication of the authors method because they used Noob</p>
<blockquote>
<p>ll data processing was performed in R (version 3.6.1; R Foundation for Statistical Computing, Vienna, Austria). Using the minfi library [26], raw IDAT files were imported; Noob normalization was performed [27]; and beta values were computed. Beta values whose corresponding intensities on probe-level failed a z-test with a P value threshold of 0.01 against background signal were masked.</p>
</blockquote>
<p><code>preprocessNoob</code> by <a href="https://doi.org/10.1093/nar/gkt090" rel="nofollow noreferrer">Triche et al (2013)</a></p>
<p>My suspicion is that <code>preprocessNoob</code> comprises the above function. I'd need to read the paper in detail.</p>
<p><code>preprocessQuantile</code> was used above and was part of the original <a href="https://doi.org/10.1093/bioinformatics/btu049" rel="nofollow noreferrer">minfi package</a></p>
<hr />
<p><strong>Note</strong> A key minfi paper describing the implementation is:
<a href="https://genomebiology.biomedcentral.com/articles/10.1186/s13059-014-0503-2" rel="nofollow noreferrer">https://genomebiology.biomedcentral.com/articles/10.1186/s13059-014-0503-2</a></p>
<p>The most advanced method I'm aware of is <a href="https://clinicalepigeneticsjournal.biomedcentral.com/articles/10.1186/s13148-019-0615-3" rel="nofollow noreferrer">Heiss and Just (2019)</a>
There guys have developed a different tool set <strong><a href="https://github.com/hhhh5/ewastools" rel="nofollow noreferrer">ewastools</a></strong>  ... if its compatible.</p>
<p>Getting involved what is a periphery topic for me at the cutting edge of the field is something I'd rather avoid, admittedly I'm heavily into cancer stuff.</p>

CC BY-SA 4.0
2024-12-14T11:54:02.480Z
@Munk Try the above but read up on Noob.
M__
CC BY-SA 4.0
2024-12-13T19:59:08.420Z
2024-12-14T12:30:49.947Z
2024-12-14T12:30:49.947Z
M__
<p>I'm trying to replicate this paper (<a href="https://pathsocjournals.onlinelibrary.wiley.com/doi/10.1002/path.5845" rel="nofollow noreferrer">https://pathsocjournals.onlinelibrary.wiley.com/doi/10.1002/path.5845</a>) and I don't understand this bit from the &quot;Data Preprocessing&quot; section on page 4:</p>
<blockquote>
<p>Beta values whose corre-sponding intensities on probe-level failed a z-test with
a P value threshold of 0.01 against background signal
were masked.</p>
</blockquote>
<p>Can someone please explain what this means? I'm using R, mainly minfi.</p>
<p>Paper details:</p>
<blockquote>
<p>Title: Machine learning models predict the primary sites of head and neck squamous cell carcinoma metastases based on DNA methylation</p>
</blockquote>
<blockquote>
<p>Details: It takes idat files from a few geo numbers for preprocessing and then models them. I have retrieved the idats from GEO used in the paper, fed them into minfi. My question is with the preprocessing while this data is in minfi.</p>
</blockquote>

CC BY-SA 4.0
2024-12-13T17:31:35.660Z
Please [edit] your question and give us more context. Ideally, make it so we only need the paper if we are particularly curious. If we need to read the paper to answer your question, that makes it far more unlikely that people will have the time to do so. So include enough for us to get the context. Also clarify exactly what part of this is confusing you. What is it you do not understand?
terdon
CC BY-SA 4.0
2024-12-14T07:15:56.633Z
@terdon added paper description to my best abilities. Forgive for I'm still a beginner
Munk
<r><bioconductor><machine-learning><methylation>
Problem while replicating methylation predicting paper, looking for explanation
# Problem while replicating methylation predicting paper, looking for explanation

(Asked by: username_0 on 2024-12-13)

I'm trying to replicate this paper (<https://pathsocjournals.onlinelibrary.wiley.com/doi/10.1002/path.5845>) and I don't understand this bit from the "Data Preprocessing" section on page 4:

> Beta values whose corre-sponding intensities on probe-level failed a z-test with
> a P value threshold of 0.01 against background signal
> were masked.

Can someone please explain what this means? I'm using R, mainly minfi.

Paper details:

> Title: Machine learning models predict the primary sites of head and neck squamous cell carcinoma metastases based on DNA methylation

> Details: It takes idat files from a few geo numbers for preprocessing and then models them. I have retrieved the idats from GEO used in the paper, fed them into minfi. My question is with the preprocessing while this data is in minfi.

**Tags:** r, bioconductor, machine-learning, methylation
> **Comments:**
> **username_1**: Please [edit] your question and give us more context. Ideally, make it so we only need the paper if we are particularly curious. If we need to read the paper to answer your question, that makes it far more unlikely that people will have the time to do so. So include enough for us to get the context. Also clarify exactly what part of this is confusing you. What is it you do not understand?  
> **username_0**: @terdon added paper description to my best abilities. Forgive for I'm still a beginner  

---

## 1 answers

### Answer by username_2 on 2024-12-13. Score: 1 | Accepted answer

Okay I see. Basically there's a threshold for removing low intensity signal that can't be distinguished from background. That threshold seems tight at 1% of the mean of the normal distributio. Only the signal that is highly significantly above background signal is retained, i.e. subject to further analysis.
The way it works is that for a given CpG site  there is a) methylated allele probe and b) unmethylated allele probe. The beta-value is a representation of those two intensities. What they want to do is filter out the background signal to avoid false positives.
Here's the manual https://bioconductor.org/packages/devel/bioc/vignettes/minfi/inst/doc/minfi.html#5_Quality_control
Here's a basic code (I stress the word basic) HOWEVER see the note on Noob in minfi below:
library(minfi)

myStuff <- read.metharray.exp("/Users/username/Desktop/myfile")
rgSet <- preprocessQuantile(myStuff)

M <- getM(rgSet)
U <- getU(rgSet)

beta <- M / (M + U)

filterProbesByZTest <- function(M, U, p.threshold = 0.01) {
  n <- ncol(M)
  filteredProbes <- rep(TRUE, ncol(M)) 
  
  for (i in 1:ncol(M)) {
    probeM <- M[, i]
    probeU <- U[, i]
    
    meanM <- mean(probeM, na.rm = TRUE)
    meanU <- mean(probeU, na.rm = TRUE)
    sdM <- sd(probeM, na.rm = TRUE)
    sdU <- sd(probeU, na.rm = TRUE)
    
    z_stat <- (meanM - meanU) / sqrt((sdM^2 / length(probeM)) + (sdU^2 / length(probeU)))
    p_value <- 2 * pnorm(-abs(z_stat))
    
    if (p_value > p.threshold) {
      filteredProbes[i] <- FALSE
    }
  }
  
  return(filteredProbes)
}

filteredProbes <- filterProbesByZTest(M, U, p.threshold = 0.01)
betaFiltered <- beta[, filteredProbes]


Note apart from being basic. the code isn't an exact replication of the authors method because they used Noob

ll data processing was performed in R (version 3.6.1; R Foundation for Statistical Computing, Vienna, Austria). Using the minfi library [26], raw IDAT files were imported; Noob normalization was performed [27]; and beta values were computed. Beta values whose corresponding intensities on probe-level failed a z-test with a P value threshold of 0.01 against background signal were masked.

preprocessNoob by Triche et al (2013)
My suspicion is that preprocessNoob comprises the above function. I'd need to read the paper in detail.
preprocessQuantile was used above and was part of the original minfi package

Note A key minfi paper describing the implementation is:
https://genomebiology.biomedcentral.com/articles/10.1186/s13059-014-0503-2
The most advanced method I'm aware of is Heiss and Just (2019)
There guys have developed a different tool set ewastools  ... if its compatible.
Getting involved what is a periphery topic for me at the cutting edge of the field is something I'd rather avoid, admittedly I'm heavily into cancer stuff.

> **Comments:**
> **username_2**: @Munk Try the above but read up on Noob.  

---

