<p>The purpose of unbiased hit rate is to avoid invalid conclusions in cases where subjects indiscriminately use one (or only few) response options. To give an example (responses in rows, stimuli in columns):</p>

<pre><code>        Happy    Sad    Angry   Fearful
Happy     1       0       0        1
Sad       7       8       6        5
Angry     0       0       0        0
Fearful   2       0       0        1
</code></pre>

<p>Consider sad faces. The uncorrected and biased hit rate for sad faces is $h_b(\mathrm{Sad})=8/8=1$. The unbiased hit rate is $h_b(\mathrm{Sad})=8^2/(8 \cdot24)=1/3$. The latter estimate better corresponds to our intuition. The subject responds almost always $\mathrm{Sad}$ and should be assigned a score slightly better than a random performance ($=1/4$).</p>

<p>You are looking for a total score. By definition a total score will account for this kind of bias as it uses all elements of the confusion matrix. Hence, there is no point in looking for a unbiased total hit rate and there is no point in trying to combine the unbiased hit rates.</p>

<p>Wagner (1993) makes it clear on page 4 that the question regarding a total score is distinct and different from the one he is trying to answer with his unbiased hit rate. In a footnote 1 on pages 27-28 he mentions methods for computing the total score:</p>

<blockquote>
  <p>There are numerous ways of examining overall performance in judgment
  studies. In psychophysics, the confusion matrix resulting from the
  method of absolute identification (as the type of
  judgment study discussed here is known) has been examined with the tools
  of information theory (see Baird &amp; Noma, 1978, for a clear exposition
  of this). [...] Other examples of measures of accuracy for  the whole
  matrix are Cohen's $\kappa$ (Cohen, 1960), and Rosenthal and Rubin's $\pi$
  (Rosenthal &amp; Rubin, 1989).</p>
</blockquote>

<p><strong>Literature</strong></p>

<p>Baird, J. C., &amp; Noma, E. J. (1978). Fundamentals of scaling and psychophysics. John Wiley &amp; Sons.</p>

<p>Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1), 37-46.</p>

<p>Rosenthal, R., &amp; Rubin, D. B. (1989). Effect size estimation for one-sample multiple-choice-type data: Design, analysis, and meta-analysis. Psychological Bulletin, 106(2), 332.</p>

CC BY-SA 3.0
2015-06-30T17:21:59.137Z
Thank you - I am also interested in the individual scores, but this provides an alternative way to examine the overall effect that should be valuable!
Joni
CC BY-SA 3.0
2015-06-30T11:41:37.610Z
2015-06-30T11:41:37.610Z
matus
<p>I am analysing data on an emotion perception task, where participants must decide if a given face is happy, sad, angry or fearful (essentially a forced choice between 4 options).</p>

<p>I am confident that I want to calculate an unbiased hit rate for each emotion (as in Wagner 1993). This takes a confusion matrix (toy example below):</p>

<pre><code>        Happy    Sad    Angry   Fearful
Happy     4       1       0        1
Sad       0       3       2        1 
Angry     0       0       6        0
Fearful   2       0       0        4
</code></pre>

<p>And the unbiased hit rate is then the correct answers squared, divided by the number of times that answer is used multiplied by the number of trials where that answer was correct. So in the above example:</p>

<pre><code>Hu(Happy) = 4^2 / ((4+0+0+2)*6) = 16/36 = 0.444
Hu(Sad) = 9/24 = 0.375
Hu(Angry) = 36/48 = 0.75
Hu(Fearful) = 16/36 = 0.444
</code></pre>

<p>These are then converted with an arcsine tranformation on the square root for use as the dependent variable in regression.</p>

<p>However, I want to use this data to assess general ability at emotion recognition in faces - what is the correct way to combine the unbiased hit rates? That is, can I generate a single value that accounts for all off these, or should I run four separate analyses and examine the overlap in the results?</p>

<p><strong>Reference</strong></p>

<p>Wagner, H. L. (1993). On measuring performance in category judgment studies of nonverbal behavior. Journal of Nonverbal Behavior, 17(1), 3-28.</p>

<emotion><methodology><categorisation><task-analysis>
Combining unbiased hit rates
Joni
# Combining unbiased hit rates

(Asked by: username_0 on 2015-06-12)

I am analysing data on an emotion perception task, where participants must decide if a given face is happy, sad, angry or fearful (essentially a forced choice between 4 options).
I am confident that I want to calculate an unbiased hit rate for each emotion (as in Wagner 1993). This takes a confusion matrix (toy example below):
        Happy    Sad    Angry   Fearful
Happy     4       1       0        1
Sad       0       3       2        1 
Angry     0       0       6        0
Fearful   2       0       0        4

And the unbiased hit rate is then the correct answers squared, divided by the number of times that answer is used multiplied by the number of trials where that answer was correct. So in the above example:
Hu(Happy) = 4^2 / ((4+0+0+2)*6) = 16/36 = 0.444
Hu(Sad) = 9/24 = 0.375
Hu(Angry) = 36/48 = 0.75
Hu(Fearful) = 16/36 = 0.444

These are then converted with an arcsine tranformation on the square root for use as the dependent variable in regression.
However, I want to use this data to assess general ability at emotion recognition in faces - what is the correct way to combine the unbiased hit rates? That is, can I generate a single value that accounts for all off these, or should I run four separate analyses and examine the overlap in the results?
Reference
Wagner, H. L. (1993). On measuring performance in category judgment studies of nonverbal behavior. Journal of Nonverbal Behavior, 17(1), 3-28.

**Tags:** emotion, methodology, categorisation, task-analysis


---

## 1 answers

### Answer by username_1 on 2015-06-30. Score: 4 | Accepted answer

The purpose of unbiased hit rate is to avoid invalid conclusions in cases where subjects indiscriminately use one (or only few) response options. To give an example (responses in rows, stimuli in columns):
        Happy    Sad    Angry   Fearful
Happy     1       0       0        1
Sad       7       8       6        5
Angry     0       0       0        0
Fearful   2       0       0        1

Consider sad faces. The uncorrected and biased hit rate for sad faces is $h_b(\mathrm{Sad})=8/8=1$. The unbiased hit rate is $h_b(\mathrm{Sad})=8^2/(8 \cdot24)=1/3$. The latter estimate better corresponds to our intuition. The subject responds almost always $\mathrm{Sad}$ and should be assigned a score slightly better than a random performance ($=1/4$).
You are looking for a total score. By definition a total score will account for this kind of bias as it uses all elements of the confusion matrix. Hence, there is no point in looking for a unbiased total hit rate and there is no point in trying to combine the unbiased hit rates.
Wagner (1993) makes it clear on page 4 that the question regarding a total score is distinct and different from the one he is trying to answer with his unbiased hit rate. In a footnote 1 on pages 27-28 he mentions methods for computing the total score:

There are numerous ways of examining overall performance in judgment
  studies. In psychophysics, the confusion matrix resulting from the
  method of absolute identification (as the type of
  judgment study discussed here is known) has been examined with the tools
  of information theory (see Baird & Noma, 1978, for a clear exposition
  of this). [...] Other examples of measures of accuracy for  the whole
  matrix are Cohen's $\kappa$ (Cohen, 1960), and Rosenthal and Rubin's $\pi$
  (Rosenthal & Rubin, 1989).

Literature
Baird, J. C., & Noma, E. J. (1978). Fundamentals of scaling and psychophysics. John Wiley & Sons.
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1), 37-46.
Rosenthal, R., & Rubin, D. B. (1989). Effect size estimation for one-sample multiple-choice-type data: Design, analysis, and meta-analysis. Psychological Bulletin, 106(2), 332.

> **Comments:**
> **username_0**: Thank you - I am also interested in the individual scores, but this provides an alternative way to examine the overall effect that should be valuable!  

---

