<h3>General thoughts on factors influencing test-retest correlations</h3>

<p>From a theoretical perspective, it makes sense that groups that experience temporary states that lower state intelligence or lead to a poorer test taking orientation would have lower test-retest cognitive test correlations. By lower state intelligence, I'm referring to states of being such as are observed with extreme sleep deprivation, alcohol intoxication, or while on psychedelic drugs that would temporarily lower actual cognitive functioning. By test taking orientation I'm referring to motivation and concentration issues related to taking a test. Because such factors are transient, they may be present in one test session and not another, and as such would be a source of error in measurement and would reduce test-retest correlations. In groups where such factors are greater or more prevalent, then test-retest correlations should be reduced more than other groups.</p>

<p>A clinician that administers a high-quality intelligence test should be aware of these transient factors when administering the test. If a test-taker is not taking a test seriously, is distracted, and so on, then the scores may have to be disregarded. Some studies might use group testing protocols that don't pick up these issues. And perhaps sometimes teasing out what is transient and what is permanent may be difficult.</p>

<p>In general, latent ability on a task is conceptualised in terms of an individual applying reasonable and normal levels of effort, under normal and healthy conditions, in a normal and reasonable environment. Under such conditions, a person is not able to improve their performance (discounting practice and cheating effects), but there are many ways that performance can decline. Sackett et al (1988) talks about the difference between typical and maximal performance. Test performance is meant to be conducted under conditions of maximal performance, where the test taker is meant to be applying maximal effort to the task.</p>

<h3>Thoughts about ADHD</h3>

<p>Research does show that in general individuals diagnosed with ADHD tend to score lower on intelligence tests. Frazier et al (2004) peformed a large-scale meta analysis and found that healthy participants scored on average 0.61 standard deviations (i.e., <a href="http://en.wikipedia.org/wiki/Effect_size#Cohen.27s_d">weighted d</a>) higher on full scale IQ than ADHD participants.</p>

<p>Many of the <a href="http://en.wikipedia.org/wiki/Attention_deficit_hyperactivity_disorder#Signs_and_symptoms">symptoms of ADHD</a>, such as being easily distracted, experiencing boredom on a task, and other problems related to sustained attention would make the administration of some cognitive tests difficult. 
It is possible for any measure of an individual's IQ to fall below what the person is capable, where such an effect can be due, in addition to random variation between testing sessions, to systematic variability in transient factors. Thus, it seems plausible that individuals with ADHD could more frequently experience such factors, and as such, experience lower test-retest correlations.</p>

<p><strong>Caveat</strong>: ADHD research is not my area. I imagine there are some  empirical studies that have compared test-retest correlations of tests between ADHD and other groups. I just haven't found them.</p>

<h3>References</h3>

<ul>
<li>Frazier, T., Demaree, H., and Youngstrom, E. (2004). Meta-analysis of
intellectual and neuropsychological test performance in
attention-deficit/hyperactivity disorder. Neuropsychology, 18(3):543.</li>
<li>Sackett, P. R., Zedeck, S., and Fogli, L. (1988). Relations between
measures of typical and maximum job performance. Journal of Applied
Psychology, 73:482-486.</li>
</ul>

CC BY-SA 3.0
2012-02-04T03:48:26.340Z
Barkley mentions a 10 point drop in IQ compared to the general population.
Indolering
CC BY-SA 3.0
2012-02-01T02:43:46.527Z
2012-02-01T02:43:46.527Z
Jeromy Anglim
<p>I've read research reporting that <a href="http://books.google.com/books?id=Au2KHXQs2JUC&amp;pg=PA59&amp;lpg=PA59&amp;dq=lower%20iq%20increased%20variability%20reaction%20time&amp;source=bl&amp;ots=f8TCnLaqmB&amp;sig=wTYAnMbuFmn_HqrS3uxXsCgCUec&amp;hl=en&amp;sa=X&amp;ei=XuwYT_uGCsamiQL22fCdCA&amp;ved=0CC0Q6AEwAQ#v=onepage&amp;q=lower%20iq%20increased%20variability%20reaction%20time&amp;f=false" rel="noreferrer">lower IQ is correlated with higher variability in reaction time</a>.  But such research does not seem to disentangle the possible effect of ADD, in which case you might have a separate cluster that does not follow the general trend.</p>

<p>So, <strong>if ADD/ADHD subjects are given a cognitive test on different days, will the variability in their outcomes be higher than that in controls?</strong></p>

CC BY-SA 3.0
2012-11-23T10:58:17.507Z
I think that to assess reaction variability you do not necessarily need to test patients twice. Usually their reaction time variability is measured within one testing session. This measure is considered a more reliable hint for diagnosing the disorder then e.g. mean reaction time differences.
jokel
CC BY-SA 3.0
2012-01-23T23:51:10.353Z
I guess the answer to this question may be related to your other question on performance variability for people with Asperger's: http://cogsci.stackexchange.com/questions/33/what-is-the-effect-of-aspergers-on-performance-variability-on-cognitive-tests-l
Jeromy Anglim
<performance><adhd><reliability>
What is the effect of ADD/ADHD on performance variability on cognitive tests over time?
InquilineKea
# What is the effect of ADD/ADHD on performance variability on cognitive tests over time?

(Asked by: username_0 on 2012-01-19)

I've read research reporting that [lower IQ is correlated with higher variability in reaction time](http://books.google.com/books?id=Au2KHXQs2JUC&pg=PA59&lpg=PA59&dq=lower%20iq%20increased%20variability%20reaction%20time&source=bl&ots=f8TCnLaqmB&sig=wTYAnMbuFmn_HqrS3uxXsCgCUec&hl=en&sa=X&ei=XuwYT_uGCsamiQL22fCdCA&ved=0CC0Q6AEwAQ#v=onepage&q=lower%20iq%20increased%20variability%20reaction%20time&f=false). But such research does not seem to disentangle the possible effect of ADD, in which case you might have a separate cluster that does not follow the general trend.

So, **if ADD/ADHD subjects are given a cognitive test on different days, will the variability in their outcomes be higher than that in controls?**

**Tags:** performance, adhd, reliability

> **Comments:**
> **username_2**: I guess the answer to this question may be related to your other question on performance variability for people with Asperger's: http://cogsci.stackexchange.com/questions/33/what-is-the-effect-of-aspergers-on-performance-variability-on-cognitive-tests-l  
> **username_1**: I think that to assess reaction variability you do not necessarily need to test patients twice. Usually their reaction time variability is measured within one testing session. This measure is considered a more reliable hint for diagnosing the disorder then e.g. mean reaction time differences.  

---

## 1 answers

### Answer by username_2 on 2012-02-01. Score: 7 | Accepted answer

### General thoughts on factors influencing test-retest correlations

From a theoretical perspective, it makes sense that groups that experience temporary states that lower state intelligence or lead to a poorer test taking orientation would have lower test-retest cognitive test correlations. By lower state intelligence, I'm referring to states of being such as are observed with extreme sleep deprivation, alcohol intoxication, or while on psychedelic drugs that would temporarily lower actual cognitive functioning. By test taking orientation I'm referring to motivation and concentration issues related to taking a test. Because such factors are transient, they may be present in one test session and not another, and as such would be a source of error in measurement and would reduce test-retest correlations. In groups where such factors are greater or more prevalent, then test-retest correlations should be reduced more than other groups.

A clinician that administers a high-quality intelligence test should be aware of these transient factors when administering the test. If a test-taker is not taking a test seriously, is distracted, and so on, then the scores may have to be disregarded. Some studies might use group testing protocols that don't pick up these issues. And perhaps sometimes teasing out what is transient and what is permanent may be difficult.

In general, latent ability on a task is conceptualised in terms of an individual applying reasonable and normal levels of effort, under normal and healthy conditions, in a normal and reasonable environment. Under such conditions, a person is not able to improve their performance (discounting practice and cheating effects), but there are many ways that performance can decline. Sackett et al (1988) talks about the difference between typical and maximal performance. Test performance is meant to be conducted under conditions of maximal performance, where the test taker is meant to be applying maximal effort to the task.

### Thoughts about ADHD

Research does show that in general individuals diagnosed with ADHD tend to score lower on intelligence tests. Frazier et al (2004) peformed a large-scale meta analysis and found that healthy participants scored on average 0.61 standard deviations (i.e., [weighted d](http://en.wikipedia.org/wiki/Effect_size#Cohen.27s_d)) higher on full scale IQ than ADHD participants.

Many of the [symptoms of ADHD](http://en.wikipedia.org/wiki/Attention_deficit_hyperactivity_disorder#Signs_and_symptoms), such as being easily distracted, experiencing boredom on a task, and other problems related to sustained attention would make the administration of some cognitive tests difficult.
It is possible for any measure of an individual's IQ to fall below what the person is capable, where such an effect can be due, in addition to random variation between testing sessions, to systematic variability in transient factors. Thus, it seems plausible that individuals with ADHD could more frequently experience such factors, and as such, experience lower test-retest correlations.

**Caveat**: ADHD research is not my area. I imagine there are some empirical studies that have compared test-retest correlations of tests between ADHD and other groups. I just haven't found them.

### References

* Frazier, T., Demaree, H., and Youngstrom, E. (2004). Meta-analysis of
  intellectual and neuropsychological test performance in
  attention-deficit/hyperactivity disorder. Neuropsychology, 18(3):543.
* Sackett, P. R., Zedeck, S., and Fogli, L. (1988). Relations between
  measures of typical and maximum job performance. Journal of Applied
  Psychology, 73:482-486.

> **Comments:**
> **username_3**: Barkley mentions a 10 point drop in IQ compared to the general population.  

---

