<p>ANOVA and linear regression are inappropriate when your outcome is a percentage of correct trials, especially so if numbers are near the boundaries 0% or 100%. </p>

<p>Mixed effects logistic regression may be appropriate. </p>

<p>More detailed answers would require more details about your experiment.  </p>

CC BY-SA 4.0
2019-05-01T13:32:46.983Z
2019-05-01T13:32:46.983Z
Bryan Krause
<p>Based on the percentage of published papers that I read, and those that I review, that use un-transformed accuracy scores, I am going to guess that is what most cogsci people do. When your accuracy is around 0.5, maybe you can get away with this, but at 0.95, I believe it is a total no go. Of the people who transform, I think they typically either convert to d' with a z-transform (inverse of the cumulative distribution of a Gaussian) or use an arcsine-square-root transformation. I am partial to the rationalized arcsine transform proposed by (<a href="https://ajslp.pubs.asha.org/doi/abs/10.1044/jshr.2803.455" rel="nofollow noreferrer">Studebaker 1985</a>) or even better using a logit mixed model (<a href="https://www.sciencedirect.com/science/article/pii/S0749596X07001337?via%3Dihub" rel="nofollow noreferrer">Jaeger 2008</a>).</p>

<p>The reason that some people think you can get away without transforming when the accuracy is around 0.5 but not around 0.9 is that the transformations (arcsine and logit) are essentially linear near 0.5 but not near 0.95.</p>

<p>Apart from ease of use and historical precedent, there really isn't a good reason not to use a logit mixed model for all statistics involving accuracy data.</p>

CC BY-SA 4.0
2019-05-01T15:25:58.787Z
Better answer than mine since you included refs. :) Is there any modern advantage to transformations rather than logistic regression for these type of data?
Bryan Krause
CC BY-SA 4.0
2019-05-01T17:36:38.523Z
@BryanKrause apart from ease of use and history, I don't think so. That said, using what your reviewers expect can be helpful. I still plot things in RAU during exploratory analyses.
StrongBad
CC BY-SA 4.0
2019-05-01T17:39:02.887Z
Ah yes, the ever present "historical reasons" method of choosing analyses. :) Thanks
Bryan Krause
CC BY-SA 4.0
2019-05-01T17:54:01.497Z
@BryanKrause that is how I run my power analysis.... The power analysis said we only needed 12 per group, but the reviewers will scream if we don't have at least 16, so we need funding to run 16.
StrongBad
CC BY-SA 4.0
2019-05-02T12:17:02.747Z
Thank you for the answers! They basically confirm my reasoning. I also found the article of Jaeger (2008) in a previous post on stackexchange.
B. Lambrev
CC BY-SA 4.0
2019-05-02T12:18:15.210Z
I have one more question though. Why if the accuracy is 0.5 I could get away with it? Is it because of the distribution?
B. Lambrev
CC BY-SA 4.0
2019-05-02T12:41:10.337Z
@B.Lambrev yes, it is the distribution, but it is easier to think about what the arcsine transformation (or logit) is doing. Around 0.5, it is just essentially linear transformation so doesn't  have much effect. Around 0.95, it isn't linear anymore.
StrongBad
CC BY-SA 4.0
2019-05-02T13:32:40.597Z
One more question: how can I convert to d' with a z-transform? Is it a transform to z-scores in order to get an effect size?
B. Lambrev
CC BY-SA 4.0
2019-05-02T13:46:23.050Z
Zscore and ztransform are different. https://en.m.wikipedia.org/wiki/Fisher_transformation @BLambrev although I'm not sure what exactly StrongBad was referring to.
Bryan Krause
CC BY-SA 4.0
2019-05-02T13:55:13.303Z
@BryanKrause I was using it to mean the inverse of the cumulative distribution of a Gaussian (aka norminv) which is usually denoted as Z(.) https://en.m.wikipedia.org/wiki/Sensitivity_index I agree the notation was confusing and thought d' would be enough. Will wed it to clarify.
StrongBad
CC BY-SA 4.0
2019-05-02T13:59:15.770Z
Ah that makes more sense in the d' context. I think I was misled by the Fisher transform being similar to the other reshape you suggested. We really need to stop overusing Z everywhere. Z transform has at least one other meaning too https://en.m.wikipedia.org/wiki/Z-transform
Bryan Krause
CC BY-SA 4.0
2019-05-02T15:19:07.343Z
Ah, ok I get it. It has nothing to do with Cohen's d. Thanks for the answers!
B. Lambrev
CC BY-SA 4.0
2019-05-02T15:48:20.430Z
@B.Lambrev yes and no. If you look at the math they are the same thing, but people talk about them differently and use the terms in different situations.
StrongBad
CC BY-SA 4.0
2019-05-02T21:01:04.430Z
@B.Lambrev Related: https://stats.stackexchange.com/questions/21399/are-cohens-d-effect-size-and-d-prime-from-the-signal-detection-theory-measuri
Bryan Krause
CC BY-SA 4.0
2019-05-01T14:59:25.050Z
2019-05-02T13:59:17.863Z
2019-05-02T13:59:17.863Z
StrongBad
<p>I have accuracy data, where every subject gives a correct, i.e. 1 or wrong, i.e. 0 answer. However every subject performs 190 trials in each session, with 10 sessions in total. </p>

<p>To analyze the data I use R, and group the results by session, i.e. more simply put every subject responds 190 times in each session, however after grouping, it gets only one value for each session (I use the group_by function of package dplyr), based on the average of all its trials in that session (could be 1.00 if all the answers are correct or 0.95, 0.85 etc. if the subject has mistakes).</p>

<p>My reasoning is that in this way the data is no longer categorical, but quantitative, therefore I was hoping to use repeated-measures ANOVA for the anlysis. However the normality tests that I did on the data showed to be signifcant and a graphical representation of the data also shows that is very strongly skewed to the right (wich is completly logical of course, since there are much more correct answers than wrong).</p>

<p>My question is: What can I use to analyze this data and what is used in the scientific community in general in those situations, since it is not an uncommon problem, accuracy being often measured? I checked some articles, but there for accuracy researchers simply report percentage differences, and state that they are (or aren't) significant.  I know that I could use Friedman's ANOVA, but it isn't that powerfull so I'm looking for another way. Also researchers often report using ANOVA for accuracy scores, which is leaving me absolutely stunned...
Maybe there are some errors in my reasoning, so all comments are welcome!!</p>

<p>Thank you for the answers!!!</p>

<cognitive-psychology><attention>
Analysis of accuracy data?
B. Lambrev
# Analysis of accuracy data?

(Asked by: username_0 on 2019-05-01)

I have accuracy data, where every subject gives a correct, i.e. 1 or wrong, i.e. 0 answer. However every subject performs 190 trials in each session, with 10 sessions in total.

To analyze the data I use R, and group the results by session, i.e. more simply put every subject responds 190 times in each session, however after grouping, it gets only one value for each session (I use the group\_by function of package dplyr), based on the average of all its trials in that session (could be 1.00 if all the answers are correct or 0.95, 0.85 etc. if the subject has mistakes).

My reasoning is that in this way the data is no longer categorical, but quantitative, therefore I was hoping to use repeated-measures ANOVA for the anlysis. However the normality tests that I did on the data showed to be signifcant and a graphical representation of the data also shows that is very strongly skewed to the right (wich is completly logical of course, since there are much more correct answers than wrong).

My question is: What can I use to analyze this data and what is used in the scientific community in general in those situations, since it is not an uncommon problem, accuracy being often measured? I checked some articles, but there for accuracy researchers simply report percentage differences, and state that they are (or aren't) significant. I know that I could use Friedman's ANOVA, but it isn't that powerfull so I'm looking for another way. Also researchers often report using ANOVA for accuracy scores, which is leaving me absolutely stunned...
Maybe there are some errors in my reasoning, so all comments are welcome!!

Thank you for the answers!!!

**Tags:** cognitive-psychology, attention


---

## 2 answers

### Answer by username_1 on 2019-05-01. Score: 2 | Accepted answer

ANOVA and linear regression are inappropriate when your outcome is a percentage of correct trials, especially so if numbers are near the boundaries 0% or 100%.

Mixed effects logistic regression may be appropriate.

More detailed answers would require more details about your experiment.


---

### Answer by username_2 on 2019-05-01. Score: 2

Based on the percentage of published papers that I read, and those that I review, that use un-transformed accuracy scores, I am going to guess that is what most cogsci people do. When your accuracy is around 0.5, maybe you can get away with this, but at 0.95, I believe it is a total no go. Of the people who transform, I think they typically either convert to d' with a z-transform (inverse of the cumulative distribution of a Gaussian) or use an arcsine-square-root transformation. I am partial to the rationalized arcsine transform proposed by ([Studebaker 1985](https://ajslp.pubs.asha.org/doi/abs/10.1044/jshr.2803.455)) or even better using a logit mixed model ([Jaeger 2008](https://www.sciencedirect.com/science/article/pii/S0749596X07001337?via%3Dihub)).

The reason that some people think you can get away without transforming when the accuracy is around 0.5 but not around 0.9 is that the transformations (arcsine and logit) are essentially linear near 0.5 but not near 0.95.

Apart from ease of use and historical precedent, there really isn't a good reason not to use a logit mixed model for all statistics involving accuracy data.

> **Comments:**
> **username_1**: Better answer than mine since you included refs. :) Is there any modern advantage to transformations rather than logistic regression for these type of data?  
> **username_2**: @BryanKrause apart from ease of use and history, I don't think so. That said, using what your reviewers expect can be helpful. I still plot things in RAU during exploratory analyses.  
> **username_1**: Ah yes, the ever present "historical reasons" method of choosing analyses. :) Thanks  
> **username_2**: @BryanKrause that is how I run my power analysis.... The power analysis said we only needed 12 per group, but the reviewers will scream if we don't have at least 16, so we need funding to run 16.  
> **username_0**: Thank you for the answers! They basically confirm my reasoning. I also found the article of Jaeger (2008) in a previous post on stackexchange.  
> **username_0**: I have one more question though. Why if the accuracy is 0.5 I could get away with it? Is it because of the distribution?  
> **username_2**: @B.Lambrev yes, it is the distribution, but it is easier to think about what the arcsine transformation (or logit) is doing. Around 0.5, it is just essentially linear transformation so doesn't  have much effect. Around 0.95, it isn't linear anymore.  
> **username_0**: One more question: how can I convert to d' with a z-transform? Is it a transform to z-scores in order to get an effect size?  
> **username_1**: Zscore and ztransform are different. https://en.m.wikipedia.org/wiki/Fisher_transformation @BLambrev although I'm not sure what exactly username_2 was referring to.  
> **username_2**: @BryanKrause I was using it to mean the inverse of the cumulative distribution of a Gaussian (aka norminv) which is usually denoted as Z(.) https://en.m.wikipedia.org/wiki/Sensitivity_index I agree the notation was confusing and thought d' would be enough. Will wed it to clarify.  
> **username_1**: Ah that makes more sense in the d' context. I think I was misled by the Fisher transform being similar to the other reshape you suggested. We really need to stop overusing Z everywhere. Z transform has at least one other meaning too https://en.m.wikipedia.org/wiki/Z-transform  
> **username_0**: Ah, ok I get it. It has nothing to do with Cohen's d. Thanks for the answers!  
> **username_2**: @B.Lambrev yes and no. If you look at the math they are the same thing, but people talk about them differently and use the terms in different situations.  
> **username_1**: @B.Lambrev Related: https://stats.stackexchange.com/questions/21399/are-cohens-d-effect-size-and-d-prime-from-the-signal-detection-theory-measuri  

---

