<p>Murphy &amp; Cleveland (1995) mention, that a good way to reduce rater errors in general is to inform raters of the existence and nature of these errors and then to simply urge to avoid them. While this reduces rater errors, it also decreases the accuracy of ratings, though. These findings come from the literature on performance assessment, where halo is usually thought of as the opposite of accuracy. The unexpected association has been termed the <em>halo-accuracy-paradox</em>. </p>

<p>Some authors have proposed that the paradoxical effect is due to different operational definitions of halo (Fisicaro, 1988). Apparently there is some evidence that the paradoxcal effect vanishes when this problem is taken care of.</p>

<p>An interesting explanation comes from Latham (Woehr &amp; Huffut, 1994). For him it's all in the way that raters are informed about halo. If raters are told that halo is a global tendency across different ratings and that it is a bad thing, then raters are going to avoid exactly that. But that does not make the ratings more accurate. So when training raters one has to be careful not to create this kind if effect.</p>

<p>In contrast, a meta-analysis by Woehr &amp; Huffcut (1994) that investigates the effectiveness of different kinds of rater trainings does not find the paradoxical effect. Instead, rater trainings moderately decreases halo and increases accuracy. The authors also found support for Latham's hypothesis. The mean effect size when rater training was in accordance with his view was bigger compared to the other cases. </p>

<p>Still, Murphy and Cleveland (1995) are quite radical in their proposal. To them, there are serious problems with all operational definitions of halo (and rater errors in general). Therefore measurements of rater errors should be abandoned altogether. Hence, in their view it doesn't make sense to speak of a paradox.</p>

<p>References:<br/>
<sub>
Fisicaro,  S. A. (1988). A reexamination of the relation between halo error and accuracy. <em>Journal of Applied Psychology.</em> 73(2), 239.<br/>
Murphy, K. R. &amp; Cleveland, J.N. (1995). <em>Understanding performance appraisal: Social, organizational, and goal-based perspectives.</em> Thousand Oaks: Sage.<br>
Woehr, D. J., &amp; Huffcutt, A. I. (1994). Rater training for performance appraisal: A quantitative review. <em>Journal of Occupational and Organizational Psychology</em>, 67(3), 189-205.
</sub></p>

CC BY-SA 3.0
2013-07-25T13:31:35.047Z
How does it reduce rater error AND decrease rating accuracy?
Krysta
CC BY-SA 3.0
2013-07-27T22:02:26.937Z
I just updated my answer with regard to your comment.
Jens Kouros
CC BY-SA 3.0
2013-07-30T11:21:04.733Z
I just stumbled upon [this very interesting blog post](http://joelcadwell.blogspot.com.es/2012/08/halo-effects-and-multicollinearity.html) that might also be relevant.
Jens Kouros
CC BY-SA 3.0
2013-08-22T14:28:55.943Z
I updated my post to include additional information that I just read about.
Jens Kouros
CC BY-SA 3.0
2013-07-23T22:36:23.087Z
2013-08-22T14:28:13.897Z
2013-08-22T14:28:13.897Z
Jens Kouros
<p>I'm working with a dataset wherein participants rate five different attributes of six device variants; the attribute ratings different variants are very tightly correlated, suggesting  that this dataset has a problem with halo error--participants form an overall impression of the quality of the device, and then instead of reassessing the device for each attribute, they answer each attribute with their overall evaluation of the device.  </p>

<p>What strategies are used to combat this effect, either before execution in the design of the study or after execution in the analysis? <strong>Citations for evidence for any strategies particularly desired.</strong></p>

CC BY-SA 3.0
2013-07-24T14:36:08.023Z
What's the process - do they look at one device variant and answer all five attributes, then move onto the next device .. or do they look at each device and rate for one attribute, and then repeat for the next attribute?
Erics
CC BY-SA 3.0
2013-07-25T13:31:02.950Z
Process modifications are possible (and one of the things I'm interested in).  They have previously looked at one device and answered all five, then moved on; I think grouping the ratings by attribute might help, but since they need to see each device and no other device while rating it, it would take a lot of square-dancing to manage and probably quite a bit more time.
Krysta
<methodology><statistics><bias><survey><halo-effect>
Minimizing Halo Error
Krysta
# Minimizing Halo Error

(Asked by: username_0 on 2013-07-23)

I'm working with a dataset wherein participants rate five different attributes of six device variants; the attribute ratings different variants are very tightly correlated, suggesting that this dataset has a problem with halo error--participants form an overall impression of the quality of the device, and then instead of reassessing the device for each attribute, they answer each attribute with their overall evaluation of the device.

What strategies are used to combat this effect, either before execution in the design of the study or after execution in the analysis? **Citations for evidence for any strategies particularly desired.**

**Tags:** methodology, statistics, bias, survey, halo-effect

> **Comments:**
> **username_1**: What's the process - do they look at one device variant and answer all five attributes, then move onto the next device .. or do they look at each device and rate for one attribute, and then repeat for the next attribute?  
> **username_0**: Process modifications are possible (and one of the things I'm interested in).  They have previously looked at one device and answered all five, then moved on; I think grouping the ratings by attribute might help, but since they need to see each device and no other device while rating it, it would take a lot of square-dancing to manage and probably quite a bit more time.  

---

## 1 answers

### Answer by username_2 on 2013-07-23. Score: 5 | Accepted answer

Murphy & Cleveland (1995) mention, that a good way to reduce rater errors in general is to inform raters of the existence and nature of these errors and then to simply urge to avoid them. While this reduces rater errors, it also decreases the accuracy of ratings, though. These findings come from the literature on performance assessment, where halo is usually thought of as the opposite of accuracy. The unexpected association has been termed the *halo-accuracy-paradox*.

Some authors have proposed that the paradoxical effect is due to different operational definitions of halo (Fisicaro, 1988). Apparently there is some evidence that the paradoxcal effect vanishes when this problem is taken care of.

An interesting explanation comes from Latham (Woehr & Huffut, 1994). For him it's all in the way that raters are informed about halo. If raters are told that halo is a global tendency across different ratings and that it is a bad thing, then raters are going to avoid exactly that. But that does not make the ratings more accurate. So when training raters one has to be careful not to create this kind if effect.

In contrast, a meta-analysis by Woehr & Huffcut (1994) that investigates the effectiveness of different kinds of rater trainings does not find the paradoxical effect. Instead, rater trainings moderately decreases halo and increases accuracy. The authors also found support for Latham's hypothesis. The mean effect size when rater training was in accordance with his view was bigger compared to the other cases.

Still, Murphy and Cleveland (1995) are quite radical in their proposal. To them, there are serious problems with all operational definitions of halo (and rater errors in general). Therefore measurements of rater errors should be abandoned altogether. Hence, in their view it doesn't make sense to speak of a paradox.

References:  
Fisicaro,  S. A. (1988). A reexamination of the relation between halo error and accuracy. *Journal of Applied Psychology.* 73(2), 239.  
Murphy, K. R. & Cleveland, J.N. (1995). *Understanding performance appraisal: Social, organizational, and goal-based perspectives.* Thousand Oaks: Sage.  
Woehr, D. J., & Huffcutt, A. I. (1994). Rater training for performance appraisal: A quantitative review. *Journal of Occupational and Organizational Psychology*, 67(3), 189-205.

> **Comments:**
> **username_0**: How does it reduce rater error AND decrease rating accuracy?  
> **username_2**: I just updated my answer with regard to your comment.  
> **username_2**: I just stumbled upon [this very interesting blog post](http://joelcadwell.blogspot.com.es/2012/08/halo-effects-and-multicollinearity.html) that might also be relevant.  
> **username_2**: I updated my post to include additional information that I just read about.  

---

