<p>Short answer: It's currently unclear.</p>
<p>In the 20+ years since the original paper was published, there have been <a href="https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect#Later_studies" rel="nofollow noreferrer">many successful replications</a> of the effect, including domains not involving written tests, such as <a href="https://doi.org/10.1016/j.obhdp.2007.05.002" rel="nofollow noreferrer">debating</a>, <a href="https://doi.org/10.1007/s11238-010-9200-0" rel="nofollow noreferrer">chess</a>, <a href="https://doi.org/10.3758/s13423-013-0379-2" rel="nofollow noreferrer">bridge</a>, <a href="https://doi.org/10.1097/00001888-200110001-00029" rel="nofollow noreferrer">bed-side manner</a>, <a href="https://doi.org/10.1016/j.cognition.2020.104345" rel="nofollow noreferrer">facial recognition</a>, and yes, even <a href="https://doi.org/10.1016/j.jsr.2009.04.006" rel="nofollow noreferrer">driving</a>.  However, as <a href="https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect" rel="nofollow noreferrer">Wikipedia</a> summarizes:</p>
<blockquote>
<p>... their conclusions are strongly challenged when subjected to
mathematical analysis and comparisons across cultures.</p>
</blockquote>
<p>That is, a significant component of the reported effect may be due to statistical artifacts that result from the method used.</p>
<p>This is not the first time that the study method used has been questioned: Theoretical criticisms appeared shortly after the original paper's publication (eg, <a href="https://doi.org/10.1037/0022-3514.82.2.180" rel="nofollow noreferrer">Krueger &amp; Mueller, 2002</a>; <a href="https://doi.org/10.1016/j.joep.2007.12.006" rel="nofollow noreferrer">Krajc &amp; Ortmann, 2008</a>; <a href="http://www.talyarkoni.org/blog/2010/07/07/What-the-Dunning-Kruger-effect-Is-and-Isnt/" rel="nofollow noreferrer">more here</a>).  However, subsequent replications of the effect correcting for these artifacts largely survived the criticisms with the main revision in results being that high performers are no longer thought to underestimate their ability.  Nonetheless, some studies that successfully replicated the effect did notice significant artifacts remaining in their empirical data - for example, <a href="https://doi.org/10.1021/ed100328c" rel="nofollow noreferrer">Bell &amp; Volckmann (2011)</a>:</p>
<blockquote>
<p>... students with the highest exam scores are just about as variable
at gauging their course knowledge as students who have the lowest exam
scores.</p>
</blockquote>
<p>Empirical evidence that other artifacts could be substantial was finally  demonstrated with the publication of a series of studies in <a href="https://doi.org/10.5038%2F1936-4660.9.1.4" rel="nofollow noreferrer">Nuhfer et al (2016)</a>, <a href="https://doi.org/10.5038%2F1936-4660.10.1.4" rel="nofollow noreferrer">Nuhfer et al (2017)</a>, who state:</p>
<blockquote>
<p>Our results show that different numerical approaches employed in
investigating and describing self-assessment accuracy are not equally
valid.</p>
</blockquote>
<p>Additional studies feature similar conclusions - for example:</p>
<p><a href="https://doi.org/10.1016/j.socec.2017.03.002" rel="nofollow noreferrer">Feld, Sauermann, &amp; de Grip (2017)</a>:</p>
<blockquote>
<p>Our results show that the unskilled are more overconfident than the
skilled.  However, as we predict in our methodological  discussion,
this relationship is significantly weaker than ordinary least squares
estimates suggest.</p>
</blockquote>
<p><a href="https://doi.org/10.1037/xge0000579" rel="nofollow noreferrer">McIntosh et al (2019)</a>:</p>
<blockquote>
<p>... the pattern can be induced (and greatly inflated) by a host of
other factors and biases, some psychologically interesting, and some
‘merely’ statistical.</p>
</blockquote>
<p><a href="https://doi.org/10.1016/j.intell.2020.101449" rel="nofollow noreferrer">Gignac &amp; Zajenkowski (2020)</a>:</p>
<blockquote>
<p>... although the phenomenon described by the Dunning-Kruger hypothesis
may be to some degree plausible for some skills, the magnitude of the
effect may be much smaller than reported previously.</p>
</blockquote>
<p>The artifacts significantly diminish the Dunning-Kruger effect size, possibly eliminating it in some cases.</p>
<p><strong>Additional notes:</strong></p>
<ul>
<li>The example in the question of people systematically overestimating their driving competency is <em>not</em> an example of the Dunning-Kruger effect.  Rather, it is an example of a different bias called the <a href="https://en.wikipedia.org/wiki/Illusory_superiority#Better-than-average_heuristic" rel="nofollow noreferrer">above-average effect</a>.</li>
<li>There is a large amount of literature on self-assessment - where subjects are asked to assess their own competence in some domain.  Across many studied domains, people tend to be somewhere around &quot;<a href="https://doi.org/10.1177%2F1745691613518075" rel="nofollow noreferrer">moderately good</a>&quot; at self-assessing their abilities, but certainly not perfect.</li>
<li>Contrary to the above-average effect, low-performers in Dunning-Kruger-type studies do <em>not</em> estimate their competence as higher than better-performing participants - rather, they rate their competence as higher than it actually is.</li>
<li>In their original paper, Dunning &amp; Kruger established a non-standard protocol for measuring the self-assessment bias, in which they divide results into sorted quartiles, and compare average absolute differences between actual and estimated test scores.  It is this protocol that is implicated in the statistical artifacts afflicting the results, as using more standard methods, such as adjusted scatter-plots, correlations, and error-bars, it does not appear as if there is much difference (if any) between low and high performers' precision.</li>
<li>Studies that adjust for statistical artifacts demonstrate that the non-standard protocol used by Dunning &amp; Kruger causes data <a href="https://en.wikipedia.org/wiki/Censoring_(statistics)" rel="nofollow noreferrer">censoring</a> (hiding the effects of noise), <a href="https://en.wikipedia.org/wiki/Ceiling_effect_(statistics)" rel="nofollow noreferrer">floor/ceiling</a> effects (error ranges are bounded), and <a href="https://en.wikipedia.org/wiki/Regression_toward_the_mean" rel="nofollow noreferrer">regression to the mean</a> (due to comparison of 2 independent measures).  Combined with some likely above-average effect, these statistical artifacts explain most (if not all) of the results.  The artifacts are so significant that even random data can generate the effect!</li>
</ul>

CC BY-SA 4.0
2021-01-25T18:15:10.830Z
Related: [What is the primary source of the “mount stupid” graphic?](https://psychology.stackexchange.com/q/17825/7001)
Arnon Weinberg
CC BY-SA 4.0
2021-01-24T05:45:09.927Z
2021-01-24T05:45:09.927Z
Arnon Weinberg
<p>Yes, there are research showing a relationship between skills and not only "sit down tests". A <a href="https://pdxscholar.library.pdx.edu/cgi/viewcontent.cgi?article=1020&amp;context=comminfolit" rel="nofollow noreferrer">systematic review</a> by Madmood (2016) found "self-reported and demonstrated information literacy skills" supported the Dunning-Kruger phenomena. Demonstrated information literacy was from derived sources such as "observations, field notes, participant reflexive journals, expert grading of searching tasks, analysis of students’ theses, quizzes, and information related assignments".</p>

<p><strong>References:</strong> </p>

<p>Mahmood, K. (2016). Do People Overestimate Their Information Literacy Skills? A Systematic Review of Empirical Evidence on the Dunning-Kruger Effect. Communications in Information Literacy, 10 (2), 199-213. <a href="https://doi.org/10.15760/comminfolit.2016.10.2.24" rel="nofollow noreferrer">https://doi.org/10.15760/comminfolit.2016.10.2.24</a></p>

CC BY-SA 4.0
2020-05-01T10:43:53.767Z
Could you elaborate on the _"self-reported and demonstrated information literacy (IL)"_? As that is the crux of the question. Maybe a simple quote from the paper where these methods are explained. For example, isn't a sit-down questionnaire 'self-reported'? Just enough to understand it without having to read the full paper. Details of course can be deferred to the paper.
Steven Jeuris
CC BY-SA 4.0
2020-05-01T05:03:28.213Z
2020-05-01T10:51:06.667Z
2020-05-01T10:51:06.667Z
Poidah
<p>The well known Dunning-Kruger effect is oft cited as a general explanation for why people with limited expertise overestimate their abilities. Other than the <a href="https://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=A501D141B334D49F8CE2492CC3A91A28?doi=10.1.1.64.2655&amp;rep=rep1&amp;type=pdf" rel="noreferrer">initial research</a> that described this cognitive bias, there seems to be little actual data to support it outside of a sit-down questionnaire or IQ test in a university setting. Driving is a well-known example (i.e. '80% of drivers consider themselves above average') although I note from <a href="https://psychology.stackexchange.com/questions/55/what-is-the-relationship-between-self-perceptions-and-actual-driving-ability">a previous post</a>, the evidence for the Dunning-Kruger effect in driving is mixed. </p>

<p><strong>Is there any data to support the Dunning-Kruger in outside of a sit-down test performed by university students?</strong></p>

<cognitive-psychology><social-psychology><methodology><experimental-psychology><bias>
Is there evidence to support the Dunning-Kruger effect?
Dr Stu
# Is there evidence to support the Dunning-Kruger effect?

(Asked by: username_0 on 2020-03-12)

The well known Dunning-Kruger effect is oft cited as a general explanation for why people with limited expertise overestimate their abilities. Other than the [initial research](https://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=A501D141B334D49F8CE2492CC3A91A28?doi=10.1.1.64.2655&rep=rep1&type=pdf) that described this cognitive bias, there seems to be little actual data to support it outside of a sit-down questionnaire or IQ test in a university setting. Driving is a well-known example (i.e. '80% of drivers consider themselves above average') although I note from [a previous post](https://psychology.stackexchange.com/questions/55/what-is-the-relationship-between-self-perceptions-and-actual-driving-ability), the evidence for the Dunning-Kruger effect in driving is mixed.

**Is there any data to support the Dunning-Kruger in outside of a sit-down test performed by university students?**

**Tags:** cognitive-psychology, social-psychology, methodology, experimental-psychology, bias


---

## 2 answers

### Answer by username_1 on 2021-01-24. Score: 3

Short answer: It's currently unclear.

In the 20+ years since the original paper was published, there have been [many successful replications](https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect#Later_studies) of the effect, including domains not involving written tests, such as [debating](https://doi.org/10.1016/j.obhdp.2007.05.002), [chess](https://doi.org/10.1007/s11238-010-9200-0), [bridge](https://doi.org/10.3758/s13423-013-0379-2), [bed-side manner](https://doi.org/10.1097/00001888-200110001-00029), [facial recognition](https://doi.org/10.1016/j.cognition.2020.104345), and yes, even [driving](https://doi.org/10.1016/j.jsr.2009.04.006). However, as [Wikipedia](https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect) summarizes:

> ... their conclusions are strongly challenged when subjected to
> mathematical analysis and comparisons across cultures.

That is, a significant component of the reported effect may be due to statistical artifacts that result from the method used.

This is not the first time that the study method used has been questioned: Theoretical criticisms appeared shortly after the original paper's publication (eg, [Krueger & Mueller, 2002](https://doi.org/10.1037/0022-3514.82.2.180); [Krajc & Ortmann, 2008](https://doi.org/10.1016/j.joep.2007.12.006); [more here](http://www.talyarkoni.org/blog/2010/07/07/What-the-Dunning-Kruger-effect-Is-and-Isnt/)). However, subsequent replications of the effect correcting for these artifacts largely survived the criticisms with the main revision in results being that high performers are no longer thought to underestimate their ability. Nonetheless, some studies that successfully replicated the effect did notice significant artifacts remaining in their empirical data - for example, [Bell & Volckmann (2011)](https://doi.org/10.1021/ed100328c):

> ... students with the highest exam scores are just about as variable
> at gauging their course knowledge as students who have the lowest exam
> scores.

Empirical evidence that other artifacts could be substantial was finally demonstrated with the publication of a series of studies in [Nuhfer et al (2016)](https://doi.org/10.5038%2F1936-4660.9.1.4), [Nuhfer et al (2017)](https://doi.org/10.5038%2F1936-4660.10.1.4), who state:

> Our results show that different numerical approaches employed in
> investigating and describing self-assessment accuracy are not equally
> valid.

Additional studies feature similar conclusions - for example:

[Feld, Sauermann, & de Grip (2017)](https://doi.org/10.1016/j.socec.2017.03.002):

> Our results show that the unskilled are more overconfident than the
> skilled. However, as we predict in our methodological discussion,
> this relationship is significantly weaker than ordinary least squares
> estimates suggest.

[McIntosh et al (2019)](https://doi.org/10.1037/xge0000579):

> ... the pattern can be induced (and greatly inflated) by a host of
> other factors and biases, some psychologically interesting, and some
> ‘merely’ statistical.

[Gignac & Zajenkowski (2020)](https://doi.org/10.1016/j.intell.2020.101449):

> ... although the phenomenon described by the Dunning-Kruger hypothesis
> may be to some degree plausible for some skills, the magnitude of the
> effect may be much smaller than reported previously.

The artifacts significantly diminish the Dunning-Kruger effect size, possibly eliminating it in some cases.

**Additional notes:**

* The example in the question of people systematically overestimating their driving competency is *not* an example of the Dunning-Kruger effect. Rather, it is an example of a different bias called the [above-average effect](https://en.wikipedia.org/wiki/Illusory_superiority#Better-than-average_heuristic).
* There is a large amount of literature on self-assessment - where subjects are asked to assess their own competence in some domain. Across many studied domains, people tend to be somewhere around "[moderately good](https://doi.org/10.1177%2F1745691613518075)" at self-assessing their abilities, but certainly not perfect.
* Contrary to the above-average effect, low-performers in Dunning-Kruger-type studies do *not* estimate their competence as higher than better-performing participants - rather, they rate their competence as higher than it actually is.
* In their original paper, Dunning & Kruger established a non-standard protocol for measuring the self-assessment bias, in which they divide results into sorted quartiles, and compare average absolute differences between actual and estimated test scores. It is this protocol that is implicated in the statistical artifacts afflicting the results, as using more standard methods, such as adjusted scatter-plots, correlations, and error-bars, it does not appear as if there is much difference (if any) between low and high performers' precision.
* Studies that adjust for statistical artifacts demonstrate that the non-standard protocol used by Dunning & Kruger causes data [censoring](https://en.wikipedia.org/wiki/Censoring_(statistics)) (hiding the effects of noise), [floor/ceiling](https://en.wikipedia.org/wiki/Ceiling_effect_(statistics)) effects (error ranges are bounded), and [regression to the mean](https://en.wikipedia.org/wiki/Regression_toward_the_mean) (due to comparison of 2 independent measures). Combined with some likely above-average effect, these statistical artifacts explain most (if not all) of the results. The artifacts are so significant that even random data can generate the effect!

> **Comments:**
> **username_1**: Related: [What is the primary source of the “mount stupid” graphic?](https://psychology.stackexchange.com/q/17825/7001)  

---

### Answer by username_2 on 2020-05-01. Score: 3

Yes, there are research showing a relationship between skills and not only "sit down tests". A [systematic review](https://pdxscholar.library.pdx.edu/cgi/viewcontent.cgi?article=1020&context=comminfolit) by Madmood (2016) found "self-reported and demonstrated information literacy skills" supported the Dunning-Kruger phenomena. Demonstrated information literacy was from derived sources such as "observations, field notes, participant reflexive journals, expert grading of searching tasks, analysis of students’ theses, quizzes, and information related assignments".

**References:**

Mahmood, K. (2016). Do People Overestimate Their Information Literacy Skills? A Systematic Review of Empirical Evidence on the Dunning-Kruger Effect. Communications in Information Literacy, 10 (2), 199-213. <https://doi.org/10.15760/comminfolit.2016.10.2.24>

> **Comments:**
> **username_3**: Could you elaborate on the _"self-reported and demonstrated information literacy (IL)"_? As that is the crux of the question. Maybe a simple quote from the paper where these methods are explained. For example, isn't a sit-down questionnaire 'self-reported'? Just enough to understand it without having to read the full paper. Details of course can be deferred to the paper.  

---

