<p>I think the relevant reliability would be the expected test-retest reliability of the measure at the two time points which are otherwise being used to examine difference scores. Thus, if you were looking at difference scores derived from an experiment where measures are taken one day apart where an intervention is applied on one of those days, then you would want to know the correlation between those two days in the absence of any intervention.</p>

<p>In general, I would question your assumption that hormone measures are more reliable than self-report personality measures. I'm not experienced with hormone reliability data, but a good personality scale should have test-retest reliability around .80 or 90. My intuition is that hormone measures have much lower reliability than this, but I'd be very interested in anyone who can cite test-retest reliability measures for various hormone measures.</p>

<p>I found this discussion of cortisol test-retest reliability (Nicolson):</p>

<blockquote>
  <p>Test-retest reliability of acute stress response measures appears to
  be low, probably because of both the underestimated noise introduced
  by spontaneous pulsatile activity (Young, Abelson, &amp; Lightman, 2004)
  and the tendency of the cortisol response to habituate following
  repeated exposures. Low reliability of cortisol outcome measures means
  that laboratory stress experiments are particularly vulnerable to Type
  2 error. This issue is important in all studies, but especially needs
  to be taken into account in intervention studies, where stress
  reactivity is compared pre- and postintervention. More analyses are
  needed to determine how many cortisol measures per session and how
  many repeated sessions are necessary to obtain reliable measures of
  stress reactivity for different subject populations and stressors
  (Gunnar &amp; Talge, 2007; Hruschka et al., 2005).</p>
</blockquote>

<h3>References</h3>

<ul>
<li>Nicolson, N. A. Measurement of Cortisol. <a href="http://www.sagepub.com/upm-data/17430_Chapter_3.pdf" rel="nofollow">http://www.sagepub.com/upm-data/17430_Chapter_3.pdf</a></li>
</ul>

CC BY-SA 3.0
2013-03-21T23:51:46.107Z
2013-03-21T23:51:46.107Z
Jeromy Anglim
<p>I was sent an email by a researcher with the following question:</p>

<blockquote>
  <p>You <a href="http://jeromyanglim.blogspot.com.au/2009/09/difference-scores-are-they-okay-to-use.html" rel="nofollow">wrote that difference scores are okay to use if measurement error
  is
  minimised</a>.</p>
  
  <ul>
  <li>Does that imply that biological data are more reliable for statistically analysing change than say a self-report personality
  scale?  </li>
  <li>For example, I've measured fear via cortisol and also self-report scale, does this mean that I can look at hormone measurements in a
  difference score but not on the self-report scales?</li>
  </ul>
</blockquote>

CC BY-SA 3.0
2016-04-08T10:45:21.900Z
Cortisol measures would not tell what kind of stressor has triggered the release - fear, sorrow, rage, allergies, physical trauma, etc.
Berit Larsen
CC BY-SA 3.0
2013-06-30T18:59:19.013Z
The following paper might be interesting: Tomarken, A.J. (1995). A psychometric perspective on psychophysiological measures. Psychological Assessment, 7 (3), 387-395.
Gala
<measurement><emotion><statistics>
Are cortisol measures more reliable than self-reported fear measures for looking at difference scores?
Jeromy Anglim
# Are cortisol measures more reliable than self-reported fear measures for looking at difference scores?

(Asked by: username_0 on 2013-03-21)

I was sent an email by a researcher with the following question:

> You [wrote that difference scores are okay to use if measurement error
> is
> minimised](http://jeromyanglim.blogspot.com.au/2009/09/difference-scores-are-they-okay-to-use.html).
>
> * Does that imply that biological data are more reliable for statistically analysing change than say a self-report personality
>   scale?
> * For example, I've measured fear via cortisol and also self-report scale, does this mean that I can look at hormone measurements in a
>   difference score but not on the self-report scales?

**Tags:** measurement, emotion, statistics

> **Comments:**
> **username_2**: The following paper might be interesting: Tomarken, A.J. (1995). A psychometric perspective on psychophysiological measures. Psychological Assessment, 7 (3), 387-395.  
> **username_1**: Cortisol measures would not tell what kind of stressor has triggered the release - fear, sorrow, rage, allergies, physical trauma, etc.  

---

## 1 answers

### Answer by username_0 on 2013-03-21. Score: 1

I think the relevant reliability would be the expected test-retest reliability of the measure at the two time points which are otherwise being used to examine difference scores. Thus, if you were looking at difference scores derived from an experiment where measures are taken one day apart where an intervention is applied on one of those days, then you would want to know the correlation between those two days in the absence of any intervention.

In general, I would question your assumption that hormone measures are more reliable than self-report personality measures. I'm not experienced with hormone reliability data, but a good personality scale should have test-retest reliability around .80 or 90. My intuition is that hormone measures have much lower reliability than this, but I'd be very interested in anyone who can cite test-retest reliability measures for various hormone measures.

I found this discussion of cortisol test-retest reliability (Nicolson):

> Test-retest reliability of acute stress response measures appears to
> be low, probably because of both the underestimated noise introduced
> by spontaneous pulsatile activity (Young, Abelson, & Lightman, 2004)
> and the tendency of the cortisol response to habituate following
> repeated exposures. Low reliability of cortisol outcome measures means
> that laboratory stress experiments are particularly vulnerable to Type
> 2 error. This issue is important in all studies, but especially needs
> to be taken into account in intervention studies, where stress
> reactivity is compared pre- and postintervention. More analyses are
> needed to determine how many cortisol measures per session and how
> many repeated sessions are necessary to obtain reliable measures of
> stress reactivity for different subject populations and stressors
> (Gunnar & Talge, 2007; Hruschka et al., 2005).

### References

* Nicolson, N. A. Measurement of Cortisol. <http://www.sagepub.com/upm-data/17430_Chapter_3.pdf>


---

