<p>MANOVA is definitely a bad idea given that one dv is continuous and the other is binomial. After exploring a number of different approaches to combining RT and accuracy data, I've come to conclude that the best current approach is to use linear ballistic accumulator model (e.g., see Donkin et al 2011). </p>

<p>The LBA is a simple (structurally and computationally) framework that lets you speak to the different processes (info processing efficiency vs response criterion) that jointly contribute to RT and error data.</p>

<h3>References</h3>

<ul>
<li>Donkin, C., Brown, S. &amp; Heathcote, A. (2011). Drawing conclusions from choice response time models: A tutorial using the linear ballistic accumulator. Journal of Mathematical Psychology, 55, 140-151. <a href="http://www.newcl.org/publications/crtmtute-revision.pdf" rel="noreferrer">PDF</a></li>
</ul>

CC BY-SA 3.0
2012-03-29T18:10:42.413Z
Thanks Mike. Why would you prefer the LBA to Ratcliff's diffusion model?
Pavel
CC BY-SA 3.0
2012-03-30T11:00:56.407Z
These days I prefer LBA because it's computationally simple/faster to fit and because I haven't seen any demonstrations of cases where diffusion can fit data that LBA can't
Mike Lawrence
CC BY-SA 3.0
2012-03-29T14:43:02.083Z
2012-10-08T10:54:32.290Z
2012-10-08T10:54:32.290Z
Mike Lawrence
<p>There are a variety of models solving accuracy and RT that have been pretty well tested and LBA is probably fine (I haven't used it).  If you don't want to go that far there is a rather simple way to analyze data controlling for SAT that has much better mathematical properties than IE scores (which, as Mike said were named by me, but offhandedly proposed by Townsend &amp; Asby, easily conceptualized as related to older rate of information scores holding information constant, and probably popularized most by Shore).</p>

<p>The first problem with the IE transformation (rt in ms ÷ proportion correct) is that it assumes a linear relationship between RT and acc.  That's clearly not the case.  While one can often achieve a linear relationship between a predictor and RT, the relationship between a predictor and accuracy is invariably an ogive.  One can make it much more linear by transforming accuracy to logit or log-odds scores (keep in mind accuracy, and in most cases even RT, are COMPLETELY arbitrary representations of what they measure).  Furthermore, rt has much better statistical properties represented as responses/second than seconds/response.  So, taking 1/rt in seconds would make that data more normal.  Therefore, logit / inverse RT scores might be a better transform.  But it's still a transform into some unknown score... I think we could call it Linearized Inverse Efficiency (or L.I.E) :)... OK, it's not actually inverse efficiency anymore since it's not an inverse rate but an accuracy corrected rate.. how about A.C.E, accuracy corrected efficiency?)</p>

<p>But... if you're going to go that far, why not just model the logistic regression on RT in each condition?  You could then hold the RT constant for each condition (maybe the grand mean) and look at the changes in predicted accuracy across conditions.  That would be a reasonable way to combine the two.</p>

<p>The only issue I've run into with that last one is that it's all about the leading edge of your RT distribution.  You need to hack off everything after accuracy asymptotes.  If what you intended to measure is the immediate response to a stimulus then that's perfectly fine.  If you want to capture something about the tails of the distributions it might not be well represented, but you could look at that separately.  You could keep that later data by just making the logistic regression quadratic.  On the flip side, one advantage you get is that you actually make use of all of the early low accuracy RTs.</p>

<p>This method does require those low accuracy RTs so you do, in general, have to encourage speed in the experiment.  That should also be done with any transform or model of RT and accuracy because you have to have some accuracy variance to work with.</p>

<p>(one thing I haven't tried, which would probably work, is just entering RT into a multilevel logistic regression of accuracy.  If you include it as an interaction term you can then examine the predicted scores holding it constant.)</p>

CC BY-SA 3.0
2012-03-30T14:19:41.233Z
Looking at the pros/cons to using a binomial mixed effects model of accuracy as a function of RT and other predictors is one of the projects I had planned before this discussion for our summer stats student, though I'm thinking of looking at both linear and generalized additive models of the effect of RT.
Mike Lawrence
CC BY-SA 3.0
2012-03-31T22:56:53.430Z
I didn't quite understand which of the mentioned approaches is the one proposed by Townsend & Ashby. The one you dubbed  L.I.E? :)
Pavel
CC BY-SA 3.0
2012-03-31T23:16:41.173Z
As for your suggestion of modeling the logistic regression on RT in each condition. Do you mean regressing response on RT (or some transformation thereof), condition and the interaction thereof? This should capture the differences in the speed-accuracy tradeoff functions, but it seems that such a model would miss differences in RT, if the SATF is the same in both conditions.
Pavel
CC BY-SA 3.0
2012-03-31T23:44:37.467Z
Pavel, the LIE is my own suggestion for a an improved implementation of the transformation suggested by T&A that is now just done as rt in ms / acc (proportion - the IE score). The logistic regression suggestion will be sensitive to rt because then you analyze the accuracy at a fixed rt using predicted values from the regression. If the rt varied but the SAT remained the same then it will be reflected in varying accuracy scores at that rt.
John
CC BY-SA 3.0
2012-03-31T23:57:02.137Z
Yes, right. I guess one could come up with theoretical situations with non-monotonic SATFs, but a logistic regression of this kind should work fine for most practical purposes, I guess. Thanks.
Pavel
CC BY-SA 3.0
2012-04-03T13:35:26.987Z
@John: Could you provide a reference to practical example where authors "modeled the logistic regression on RT in each condition"? I would really appreciate that.
Geek On Acid
CC BY-SA 3.0
2012-04-03T23:43:40.320Z
I don't know of one.  There weren't any using I.E. scores until Enns first used it I believe.  I used it in my thesis after that and called it I.E. scores.  Why not be the first?
John
CC BY-SA 3.0
2012-03-30T12:39:15.957Z
2012-06-28T08:07:43.113Z
2012-06-28T08:07:43.113Z
John
<p>Another possible approach is using EZ-diffusion model suggested by <a href="http://www.ejwagenmakers.com/2007/EZ.pdf">Wagenmaker, van der Mass and Grasman (2007)</a>. Quoting <a href="http://psych.stanford.edu/~jlm/pdfs/BrownHeathcote08.pdf">Brown &amp; Heathcote (2008; p. 4)</a>: </p>

<blockquote>
  <p>This model is extremely simple, with just one source of variability in
  evidence accumulation—within-trial randomness—and simple linear
  accumulation (although evidence for one response does count against
  the other). The EZ-diffusion model is even simpler than the LBA, but
  it is incomplete. Wagen- makers et al. proposed the EZ-diffusion as a
  descriptive rather than process model, with the aim of adequately
  describing data as simply as possible. The tradeoff in developing such
  a simple model was that it could not account for some of the empirical
  phenomena in choice RT, such as the relative speed of correct vs.
  incorrect responses.</p>
</blockquote>

CC BY-SA 3.0
2012-04-09T13:14:04.070Z
2012-04-09T13:14:04.070Z
Geek On Acid
<p>I recently had similar problem and I used inverse efficiency (IE) scores. These scores
were derived by dividing the response times by correct response rates separately for each condition, carried out in such a way that the higher the score was, the worse was the performance. So you get something like "corrected reaction time" scores. Here is example of paper that uses it - check Experiment 2 on page 144:</p>

<blockquote>
  <p>Petrini, K., McAleer, P., &amp; Pollick, F. E. (2010). <a href="http://www.psy.gla.ac.uk/docs/download.php?type=PUBLS&amp;id=1482">Audiovisual
  integration of emotional signals from music improvisation does not
  depend on temporal correspondence</a>. <em>Brain research</em>, <em>1323</em>,
  139-148.</p>
</blockquote>

CC BY-SA 3.0
2012-03-30T11:06:51.620Z
Unfortunately inverse efficiency assumes a particular scaling of RT and error rate that is completely unsupported. Even Dr. John Christie, a colleague of mine that gave IE its name, has since come to completely oppose its use.
Mike Lawrence
CC BY-SA 3.0
2012-03-30T11:24:56.993Z
Good to know @Mike, thank you. I will give IBA a shot when I get my head around it. It's interesting tho, that in few papers investigating multisensory integration of emotional information (paper above, and [this other paper](http://www.sciencedirect.com/science/article/pii/S0006899308009372)) you can find the use of IE as a standard procedure...
Geek On Acid
CC BY-SA 3.0
2012-03-30T12:16:27.443Z
IE has become lamentably standard in some fields partly, I suspect, because it feels somewhat intuitive and is certainly an easy "solution" to the vexing problem of combining speed and accuracy. Admittedly, I don't have data explicitly demonstrating the invalidity of the speed-accuracy scaling assumed by IE (though I now think I'll task a student to generate this data through simulations this summer), but it would seem rather remarkable if the simple IE scaling, which at least rather questionably models accuracy as a proportion (see Dixon, 2008, "Models of accuracy..."), comes out as valid.
Mike Lawrence
CC BY-SA 3.0
2012-03-30T12:24:41.153Z
I think that, after having been in the presence of numerous people using it, including myself, it's become common because it can make you effects bigger and make the very difficult to explain SAT go away.  Neither of those are great reasons to do anything.
John
CC BY-SA 3.0
2012-03-29T22:56:56.380Z
2012-03-29T22:56:56.380Z
Geek On Acid
<p>As mentioned in the other comments, ANOVA is problematic when mixing types of predictor variables. (Generalized) mixed effects models are gaining popularity these days and actually provide a very convenient way for modelling such things. A paper demonstrating the efficacy of this approach as well as giving a tutorial-like introduction is:</p>

<p>Davidson, D. J. und A. E. Martin (2013). Modeling accuracy as a function of response time with the generalized linear mixed effects model. Acta Psychologica, 144:83– 96.</p>

<p>There's also a <a href="https://stats.stackexchange.com/questions/25548/analysis-of-speed-accuracy-tradeoff-in-lmer">related post</a> on CrossValidated.</p>

CC BY-SA 3.0
2014-04-18T16:21:42.683Z
2014-04-18T22:39:39.597Z
2017-04-13T12:44:18.443Z
Livius
<p>A paper comparing the performance of Inverse Efficiency Scores and diffusion models for the quantification of RT and accuracy can be found <a href="https://link.springer.com/article/10.1007/s00426-010-0289-0" rel="nofollow noreferrer">here</a>.</p>
<p>Rach et al. (2011) &quot;On quantifying multisensory interaction effects in reaction time and detection rate &quot; Psychological Research
Volume 75, Number 2, 77-94, <a href="https://doi.org/10.1007/s00426-010-0289-0" rel="nofollow noreferrer">DOI</a>, <a href="https://uol.de/f/6/dept/psycho/ag/kogn/colonius/rach_diederich_colonius_OnQuantifyingMultisensoryInteractionPsychRes2010.pdf" rel="nofollow noreferrer">PDF</a></p>

CC BY-SA 4.0
2012-04-11T09:15:29.047Z
2022-06-06T17:15:07.467Z
2022-06-06T17:15:07.467Z
H.Muster
<p>Is there a good way to analyze reaction times and accuracy together, other than <a href="http://en.wikipedia.org/wiki/Multivariate_analysis_of_variance">MANOVA</a>?</p>

<p>I have data from an experiment in which participants had to respond to stimuli in two different conditions in a within-subject design. And my impression is that some subjects differ in accuracy between those two conditions, while some others differ in reaction time. 
<strong>How can I test such a hypothesis?</strong></p>

CC BY-SA 3.0
2012-03-29T14:57:40.253Z
You might also want to take a look at [this question](http://cogsci.stackexchange.com/q/541/29) for inspiration.
Artem Kaznatcheev
<methodology><experimental-psychology><statistics><reaction-time>
How to analyze reaction times and accuracy together?
Pavel
# How to analyze reaction times and accuracy together?

(Asked by: username_0 on 2012-03-29)

Is there a good way to analyze reaction times and accuracy together, other than [MANOVA](http://en.wikipedia.org/wiki/Multivariate_analysis_of_variance)?

I have data from an experiment in which participants had to respond to stimuli in two different conditions in a within-subject design. And my impression is that some subjects differ in accuracy between those two conditions, while some others differ in reaction time.
**How can I test such a hypothesis?**

**Tags:** methodology, experimental-psychology, statistics, reaction-time

> **Comments:**
> **username_1**: You might also want to take a look at [this question](http://cogsci.stackexchange.com/q/541/29) for inspiration.  

---

## 6 answers

### Answer by username_6 on 2012-04-11. Score: 13 | Accepted answer

A paper comparing the performance of Inverse Efficiency Scores and diffusion models for the quantification of RT and accuracy can be found [here](https://link.springer.com/article/10.1007/s00426-010-0289-0).

Rach et al. (2011) "On quantifying multisensory interaction effects in reaction time and detection rate " Psychological Research
Volume 75, Number 2, 77-94, [DOI](https://doi.org/10.1007/s00426-010-0289-0), [PDF](https://uol.de/f/6/dept/psycho/ag/kogn/colonius/rach_diederich_colonius_OnQuantifyingMultisensoryInteractionPsychRes2010.pdf)


---

### Answer by username_2 on 2012-03-29. Score: 15

MANOVA is definitely a bad idea given that one dv is continuous and the other is binomial. After exploring a number of different approaches to combining RT and accuracy data, I've come to conclude that the best current approach is to use linear ballistic accumulator model (e.g., see Donkin et al 2011).

The LBA is a simple (structurally and computationally) framework that lets you speak to the different processes (info processing efficiency vs response criterion) that jointly contribute to RT and error data.

### References

* Donkin, C., Brown, S. & Heathcote, A. (2011). Drawing conclusions from choice response time models: A tutorial using the linear ballistic accumulator. Journal of Mathematical Psychology, 55, 140-151. [PDF](http://www.newcl.org/publications/crtmtute-revision.pdf)

> **Comments:**
> **username_0**: Thanks Mike. Why would you prefer the LBA to Ratcliff's diffusion model?  
> **username_2**: These days I prefer LBA because it's computationally simple/faster to fit and because I haven't seen any demonstrations of cases where diffusion can fit data that LBA can't  

---

### Answer by username_3 on 2012-03-30. Score: 10

There are a variety of models solving accuracy and RT that have been pretty well tested and LBA is probably fine (I haven't used it). If you don't want to go that far there is a rather simple way to analyze data controlling for SAT that has much better mathematical properties than IE scores (which, as Mike said were named by me, but offhandedly proposed by Townsend & Asby, easily conceptualized as related to older rate of information scores holding information constant, and probably popularized most by Shore).

The first problem with the IE transformation (rt in ms ÷ proportion correct) is that it assumes a linear relationship between RT and acc. That's clearly not the case. While one can often achieve a linear relationship between a predictor and RT, the relationship between a predictor and accuracy is invariably an ogive. One can make it much more linear by transforming accuracy to logit or log-odds scores (keep in mind accuracy, and in most cases even RT, are COMPLETELY arbitrary representations of what they measure). Furthermore, rt has much better statistical properties represented as responses/second than seconds/response. So, taking 1/rt in seconds would make that data more normal. Therefore, logit / inverse RT scores might be a better transform. But it's still a transform into some unknown score... I think we could call it Linearized Inverse Efficiency (or L.I.E) :)... OK, it's not actually inverse efficiency anymore since it's not an inverse rate but an accuracy corrected rate.. how about A.C.E, accuracy corrected efficiency?)

But... if you're going to go that far, why not just model the logistic regression on RT in each condition? You could then hold the RT constant for each condition (maybe the grand mean) and look at the changes in predicted accuracy across conditions. That would be a reasonable way to combine the two.

The only issue I've run into with that last one is that it's all about the leading edge of your RT distribution. You need to hack off everything after accuracy asymptotes. If what you intended to measure is the immediate response to a stimulus then that's perfectly fine. If you want to capture something about the tails of the distributions it might not be well represented, but you could look at that separately. You could keep that later data by just making the logistic regression quadratic. On the flip side, one advantage you get is that you actually make use of all of the early low accuracy RTs.

This method does require those low accuracy RTs so you do, in general, have to encourage speed in the experiment. That should also be done with any transform or model of RT and accuracy because you have to have some accuracy variance to work with.

(one thing I haven't tried, which would probably work, is just entering RT into a multilevel logistic regression of accuracy. If you include it as an interaction term you can then examine the predicted scores holding it constant.)

> **Comments:**
> **username_2**: Looking at the pros/cons to using a binomial mixed effects model of accuracy as a function of RT and other predictors is one of the projects I had planned before this discussion for our summer stats student, though I'm thinking of looking at both linear and generalized additive models of the effect of RT.  
> **username_0**: I didn't quite understand which of the mentioned approaches is the one proposed by Townsend & Ashby. The one you dubbed  L.I.E? :)  
> **username_0**: As for your suggestion of modeling the logistic regression on RT in each condition. Do you mean regressing response on RT (or some transformation thereof), condition and the interaction thereof? This should capture the differences in the speed-accuracy tradeoff functions, but it seems that such a model would miss differences in RT, if the SATF is the same in both conditions.  
> **username_3**: username_0, the LIE is my own suggestion for a an improved implementation of the transformation suggested by T&A that is now just done as rt in ms / acc (proportion - the IE score). The logistic regression suggestion will be sensitive to rt because then you analyze the accuracy at a fixed rt using predicted values from the regression. If the rt varied but the SAT remained the same then it will be reflected in varying accuracy scores at that rt.  
> **username_0**: Yes, right. I guess one could come up with theoretical situations with non-monotonic SATFs, but a logistic regression of this kind should work fine for most practical purposes, I guess. Thanks.  
> **username_4**: @username_3: Could you provide a reference to practical example where authors "modeled the logistic regression on RT in each condition"? I would really appreciate that.  
> **username_3**: I don't know of one.  There weren't any using I.E. scores until Enns first used it I believe.  I used it in my thesis after that and called it I.E. scores.  Why not be the first?  

---

### Answer by username_4 on 2012-04-09. Score: 7

Another possible approach is using EZ-diffusion model suggested by [Wagenmaker, van der Mass and Grasman (2007)](http://www.ejwagenmakers.com/2007/EZ.pdf). Quoting [Brown & Heathcote (2008; p. 4)](http://psych.stanford.edu/~jlm/pdfs/BrownHeathcote08.pdf):

> This model is extremely simple, with just one source of variability in
> evidence accumulation—within-trial randomness—and simple linear
> accumulation (although evidence for one response does count against
> the other). The EZ-diffusion model is even simpler than the LBA, but
> it is incomplete. Wagen- makers et al. proposed the EZ-diffusion as a
> descriptive rather than process model, with the aim of adequately
> describing data as simply as possible. The tradeoff in developing such
> a simple model was that it could not account for some of the empirical
> phenomena in choice RT, such as the relative speed of correct vs.
> incorrect responses.


---

### Answer by username_4 on 2012-03-29. Score: 6

I recently had similar problem and I used inverse efficiency (IE) scores. These scores
were derived by dividing the response times by correct response rates separately for each condition, carried out in such a way that the higher the score was, the worse was the performance. So you get something like "corrected reaction time" scores. Here is example of paper that uses it - check Experiment 2 on page 144:

> Petrini, K., McAleer, P., & Pollick, F. E. (2010). [Audiovisual
> integration of emotional signals from music improvisation does not
> depend on temporal correspondence](http://www.psy.gla.ac.uk/docs/download.php?type=PUBLS&id=1482). *Brain research*, *1323*,
> 139-148.

> **Comments:**
> **username_2**: Unfortunately inverse efficiency assumes a particular scaling of RT and error rate that is completely unsupported. Even Dr. username_3 Christie, a colleague of mine that gave IE its name, has since come to completely oppose its use.  
> **username_4**: Good to know @Mike, thank you. I will give IBA a shot when I get my head around it. It's interesting tho, that in few papers investigating multisensory integration of emotional information (paper above, and [this other paper](http://www.sciencedirect.com/science/article/pii/S0006899308009372)) you can find the use of IE as a standard procedure...  
> **username_2**: IE has become lamentably standard in some fields partly, I suspect, because it feels somewhat intuitive and is certainly an easy "solution" to the vexing problem of combining speed and accuracy. Admittedly, I don't have data explicitly demonstrating the invalidity of the speed-accuracy scaling assumed by IE (though I now think I'll task a student to generate this data through simulations this summer), but it would seem rather remarkable if the simple IE scaling, which at least rather questionably models accuracy as a proportion (see Dixon, 2008, "Models of accuracy..."), comes out as valid.  
> **username_3**: I think that, after having been in the presence of numerous people using it, including myself, it's become common because it can make you effects bigger and make the very difficult to explain SAT go away.  Neither of those are great reasons to do anything.  

---

### Answer by username_5 on 2014-04-18. Score: 4

As mentioned in the other comments, ANOVA is problematic when mixing types of predictor variables. (Generalized) mixed effects models are gaining popularity these days and actually provide a very convenient way for modelling such things. A paper demonstrating the efficacy of this approach as well as giving a tutorial-like introduction is:

Davidson, D. J. und A. E. Martin (2013). Modeling accuracy as a function of response time with the generalized linear mixed effects model. Acta Psychologica, 144:83– 96.

There's also a [related post](https://stats.stackexchange.com/questions/25548/analysis-of-speed-accuracy-tradeoff-in-lmer) on CrossValidated.


---

