<p>The paper doesn't seem to have been written for general readers. Unfortunately I don't understand the equations enough to be able to explain what the categorical and continuous variables are. Just in case it helps, here are the equations from the paper showing the precision/recall calculations:</p>

<p>$$\begin{array}{@{}rcl@{}} \text{pr}(\tau) &amp;=&amp; \frac{1}{m(\tau)}\sum\limits_{i=1}^{m(\tau)} \frac{{\sum\nolimits}_{f} \unicode{x1D7D9}\left(f \in P_{i}(\tau) \wedge f \in T_{i}\right)}{\sum_{f} \unicode{x1D7D9}\left(f \in P_{i}(\tau) \right)},\\ \text{rc}(\tau) &amp;=&amp; \frac{1}{n_{e}}\sum\limits_{i=1}^{n_{e}} \frac{{\sum\nolimits}_{f} \unicode{x1D7D9}\left(f \in {P}_{i}(\tau) \wedge f \in T_{i}\right)}{{\sum\nolimits}_{f} \unicode{x1D7D9}\left(f \in T_{i} \right)}, \\ F_{\max} &amp;=&amp; \max_{\tau} \left\{ \frac{2\cdot \text{pr}(\tau)\cdot \text{rc}(\tau)}{\text{pr}(\tau) + \text{rc}(\tau)} \right\}, \end{array} $$</p>

<blockquote>
  <p>where P i (τ) denotes the set of terms that have predicted scores greater than or equal to τ for a protein sequence i, T i denotes the corresponding ground-truth set of terms for that sequence, m(τ) is the number of sequences with at least one predicted score greater than or equal to τ, $\unicode{x1D7D9}\left (\cdot \right)$ is an indicator function, and n e is the number of targets used in a particular mode of evaluation.</p>
</blockquote>

CC BY-SA 3.0
2018-01-04T08:37:39.840Z
2018-01-04T08:37:39.840Z
gringer
<p>From a quick reading of the paper it looks like they are predicting biological function of proteins. They frame this as a classification problem, where they try to classify a protein as pertaining to a set of terms (Abnormality of the lymph nodes, Abnormality of skin physiology, etc). It looks like they predict for each protein whether it falls into this category and create a ROC curve to plot the false positive and true positive rates of prediction.</p>

<p>In this case, the continuous variable would be your predictions (Abnormality of the lymph nodes: predicted 0.65 probability, Abnormality of skin physiology: predicted 0.25 probability). The categorical variable would then be the ground truth that you are trying to predict (Abnormality of the lymph nodes: known 1, Abnormality of skin physiology: known 0). Based on these two variables you can make your ROC curve.</p>

CC BY-SA 3.0
2018-03-06T21:18:20.577Z
2018-03-06T21:18:20.577Z
Sevy
<p>My question is about ROC curves used in CAFA2 experiment. <a href="https://genomebiology.biomedcentral.com/articles/10.1186/s13059-016-1037-6" rel="nofollow noreferrer">In this paper</a> they used the ROC analysis for the term-centric evaluation. In order to 
perform ROC curve analysis we should have a continuous variable and a classifier (categorical) variable. In CAFA2 experiment, 
which variables did they consider as a categorical and continuous? Thank you.</p>

<proteins>
ROC analysis, CAFA 2 experiment
# ROC analysis, CAFA 2 experiment

(Asked by: username_0 on 2018-01-04)

My question is about ROC curves used in CAFA2 experiment. [In this paper](https://genomebiology.biomedcentral.com/articles/10.1186/s13059-016-1037-6) they used the ROC analysis for the term-centric evaluation. In order to
perform ROC curve analysis we should have a continuous variable and a classifier (categorical) variable. In CAFA2 experiment,
which variables did they consider as a categorical and continuous? Thank you.

**Tags:** proteins

---

## 2 answers

### Answer by username_1 on 2018-01-04. Score: 2

The paper doesn't seem to have been written for general readers. Unfortunately I don't understand the equations enough to be able to explain what the categorical and continuous variables are. Just in case it helps, here are the equations from the paper showing the precision/recall calculations:

$$\begin{array}{@{}rcl@{}} \text{pr}(\tau) &=& \frac{1}{m(\tau)}\sum\limits\_{i=1}^{m(\tau)} \frac{{\sum\nolimits}\_{f} \unicode{x1D7D9}\left(f \in P\_{i}(\tau) \wedge f \in T\_{i}\right)}{\sum\_{f} \unicode{x1D7D9}\left(f \in P\_{i}(\tau) \right)},\\ \text{rc}(\tau) &=& \frac{1}{n\_{e}}\sum\limits\_{i=1}^{n\_{e}} \frac{{\sum\nolimits}\_{f} \unicode{x1D7D9}\left(f \in {P}\_{i}(\tau) \wedge f \in T\_{i}\right)}{{\sum\nolimits}\_{f} \unicode{x1D7D9}\left(f \in T\_{i} \right)}, \\ F\_{\max} &=& \max\_{\tau} \left\{ \frac{2\cdot \text{pr}(\tau)\cdot \text{rc}(\tau)}{\text{pr}(\tau) + \text{rc}(\tau)} \right\}, \end{array} $$

> where P i (τ) denotes the set of terms that have predicted scores greater than or equal to τ for a protein sequence i, T i denotes the corresponding ground-truth set of terms for that sequence, m(τ) is the number of sequences with at least one predicted score greater than or equal to τ, $\unicode{x1D7D9}\left (\cdot \right)$ is an indicator function, and n e is the number of targets used in a particular mode of evaluation.


---

### Answer by username_2 on 2018-03-06. Score: 2

From a quick reading of the paper it looks like they are predicting biological function of proteins. They frame this as a classification problem, where they try to classify a protein as pertaining to a set of terms (Abnormality of the lymph nodes, Abnormality of skin physiology, etc). It looks like they predict for each protein whether it falls into this category and create a ROC curve to plot the false positive and true positive rates of prediction.

In this case, the continuous variable would be your predictions (Abnormality of the lymph nodes: predicted 0.65 probability, Abnormality of skin physiology: predicted 0.25 probability). The categorical variable would then be the ground truth that you are trying to predict (Abnormality of the lymph nodes: known 1, Abnormality of skin physiology: known 0). Based on these two variables you can make your ROC curve.


---

