<p>Even among researchers there is widespread misunderstanding of core statistics ideas. Look at the work by Geoff Cumming. Example paper title: <a href="http://irt.com.ne.kr/data/researchers%20misunderstand%20ci%20and%20error%20bars.pdf">'Researchers misunderstand confidence intervals and standard error bars.'</a></p>

CC BY-SA 3.0
2015-04-14T13:22:03.707Z
2015-04-14T13:24:03.883Z
2015-04-14T13:24:03.883Z
Tom
<p>To add to Tom's answer and expand on my comment that lay people and researchers both generally have an inadequate understanding of basic statistics, another study by Hoekstra, Morey, Rouder and Wagenmakers (2014) asked 120 researchers and grad students plus 442 first-year students in psychology to indicate whether the following six different interpretations of a 95% confidence interval were correct.</p>
<blockquote>
<p><strong>Questionnaire content</strong> <a href="http://psychology.okstate.edu/faculty/jgrice/psyc5314/CI_Interpretation.pdf" rel="nofollow noreferrer">(Hoekstra, Morey, Rouder and Wagenmakers, 2014)</a></p>
<ol>
<li>The probability that the true mean is greater than 0 is at least 95 %.</li>
<li>The probability that the true mean equals 0 is smaller than 5 %.</li>
<li>The “null hypothesis” that the true mean equals 0 is likely to be incorrect.</li>
<li>There is a 95 % probability that the true mean lies between 0.1 and 0.4.</li>
<li>We can be 95 % confident that the true mean lies between 0.1 and 0.4.</li>
<li>If we were to repeat the experiment over and over, then 95 % of the time the true mean falls between 0.1 and 0.4.</li>
</ol>
<p><sub>(I highly recommend looking up the quite humorous questionnaire, featuring Professor Bumbledorf, in Appendix 2.</sub></p>
</blockquote>
<p>On average, participants endorsed more than three of these statements, and all three subgroups gave substantial endorsement for individual statements. <strong>All of the statements are wrong, however.</strong>  Only eight first-year students and three researchers gave a flawless response, i.e., all false, despite the questionnaire explicitly raising this possibility!</p>
<blockquote class="spoiler">
<p> The correct interpretation is, “If we were to repeat the experiment over and over, then the confidence intervals contain the true mean 95% of the time.”</p>
</blockquote>
<p><sub>(Mouse-over for the correct interpretation.)</sub></p>
<h3>References</h3>
<ul>
<li><a href="http://psychology.okstate.edu/faculty/jgrice/psyc5314/CI_Interpretation.pdf" rel="nofollow noreferrer">Hoekstra, R., Morey, R. D., Rouder, J. N., &amp; Wagenmakers, E. J. (2014). Robust misinterpretation of confidence intervals. Psychonomic bulletin &amp; review, 21(5), 1157-1164.</a></li>
</ul>

CC BY-SA 3.0
2015-05-10T10:52:48.493Z
Is there anything about how statistics in psychology are taught differently in various places? I noticed this was at the univ. of Amsterdam. I did my BSc in the UK and we aren't really taught statistics, more how to use SPSS
queenslug
CC BY-SA 3.0
2015-05-10T11:01:22.763Z
@queenslug Anecdotally, I would say that sums up most undergrad statistics. (But don't get me started.) Also, given that many of the researchers tested were not trained at the UvA, only working there, I would say there's every reason to suspect that it's fairly representative.
Christian Hummeluhr
CC BY-SA 3.0
2015-05-10T09:21:20.427Z
2015-05-10T10:05:12.357Z
2020-06-17T09:55:31.633Z
Christian Hummeluhr
<p>I'm looking for references related to how researchers and laypeople understand concepts of statistical inference. In particular, I'm looking for articles in which subjects could demonstrate statistical competence by evaluating statistical claims. Does anyone have a relevant reference?</p>

CC BY-SA 3.0
2015-04-14T13:19:23.167Z
Even among researchers there is widespread misunderstanding of core statistics ideas. Look at the work by Geoff Cumming. Example paper title: 'Researchers misunderstand confidence intervals and standard error bars.'
Tom
CC BY-SA 3.0
2015-04-14T13:20:56.703Z
Thanks, this is exactly what I'm looking for. I would certainly upvote your comment as an answer.
Josh
CC BY-SA 3.0
2015-04-15T10:23:30.777Z
[This wonderful TED talk](https://www.youtube.com/watch?v=kLmzxmRcUTo) may be of interest.
lemon
CC BY-SA 3.0
2015-04-16T09:58:31.800Z
The general consensus in statistical psychology is that lay people and researchers understand statistics in the same way, generally speaking, and that way is "poorly." (Joke disclaimer.)
Christian Hummeluhr
CC BY-SA 3.0
2015-05-10T11:20:54.790Z
@tom good point. another one i love is the statement "approached statistical significance" ... moreover, i recall reading a review of journals that showed very few studies having reported testing for normality in parametric analyses. i also wish more studies would report effect size. it's quite funny how research publication obliges one to stick to the statistical reporting conventions of the field/journal, no matter how inappropriate such conventions may be.
faustus
CC BY-SA 3.0
2015-05-10T17:16:56.260Z
@faustus "Funny" is not the term I usually use to describe the situation. I prefer "farcical," but would also accept "rage-inducing" or "soul-crushing."
Christian Hummeluhr
<cognitive-psychology><reference-request>
Do researchers and laypeople understand statistics differently?
Josh
# Do researchers and laypeople understand statistics differently?

(Asked by: username_0 on 2015-04-13)

I'm looking for references related to how researchers and laypeople understand concepts of statistical inference. In particular, I'm looking for articles in which subjects could demonstrate statistical competence by evaluating statistical claims. Does anyone have a relevant reference?

**Tags:** cognitive-psychology, reference-request

> **Comments:**
> **username_1**: Even among researchers there is widespread misunderstanding of core statistics ideas. Look at the work by Geoff Cumming. Example paper title: 'Researchers misunderstand confidence intervals and standard error bars.'  
> **username_0**: Thanks, this is exactly what I'm looking for. I would certainly upvote your comment as an answer.  
> **username_2**: [This wonderful TED talk](https://www.youtube.com/watch?v=kLmzxmRcUTo) may be of interest.  
> **username_3**: The general consensus in statistical psychology is that lay people and researchers understand statistics in the same way, generally speaking, and that way is "poorly." (Joke disclaimer.)  
> **username_4**: @tom good point. another one i love is the statement "approached statistical significance" ... moreover, i recall reading a review of journals that showed very few studies having reported testing for normality in parametric analyses. i also wish more studies would report effect size. it's quite funny how research publication obliges one to stick to the statistical reporting conventions of the field/journal, no matter how inappropriate such conventions may be.  
> **username_3**: @username_4 "Funny" is not the term I usually use to describe the situation. I prefer "farcical," but would also accept "rage-inducing" or "soul-crushing."  

---

## 2 answers

### Answer by username_1 on 2015-04-14. Score: 9

Even among researchers there is widespread misunderstanding of core statistics ideas. Look at the work by Geoff Cumming. Example paper title: ['Researchers misunderstand confidence intervals and standard error bars.'](http://irt.com.ne.kr/data/researchers%20misunderstand%20ci%20and%20error%20bars.pdf)


---

### Answer by username_3 on 2015-05-10. Score: 3

To add to username_1's answer and expand on my comment that lay people and researchers both generally have an inadequate understanding of basic statistics, another study by Hoekstra, Morey, Rouder and Wagenmakers (2014) asked 120 researchers and grad students plus 442 first-year students in psychology to indicate whether the following six different interpretations of a 95% confidence interval were correct.

> **Questionnaire content** [(Hoekstra, Morey, Rouder and Wagenmakers, 2014)](http://psychology.okstate.edu/faculty/jgrice/psyc5314/CI_Interpretation.pdf)
>
> 1. The probability that the true mean is greater than 0 is at least 95 %.
> 2. The probability that the true mean equals 0 is smaller than 5 %.
> 3. The “null hypothesis” that the true mean equals 0 is likely to be incorrect.
> 4. There is a 95 % probability that the true mean lies between 0.1 and 0.4.
> 5. We can be 95 % confident that the true mean lies between 0.1 and 0.4.
> 6. If we were to repeat the experiment over and over, then 95 % of the time the true mean falls between 0.1 and 0.4.
>
> (I highly recommend looking up the quite humorous questionnaire, featuring Professor Bumbledorf, in Appendix 2.

On average, participants endorsed more than three of these statements, and all three subgroups gave substantial endorsement for individual statements. **All of the statements are wrong, however.** Only eight first-year students and three researchers gave a flawless response, i.e., all false, despite the questionnaire explicitly raising this possibility!

> The correct interpretation is, “If we were to repeat the experiment over and over, then the confidence intervals contain the true mean 95% of the time.”

(Mouse-over for the correct interpretation.)

### References

* [Hoekstra, R., Morey, R. D., Rouder, J. N., & Wagenmakers, E. J. (2014). Robust misinterpretation of confidence intervals. Psychonomic bulletin & review, 21(5), 1157-1164.](http://psychology.okstate.edu/faculty/jgrice/psyc5314/CI_Interpretation.pdf)

> **Comments:**
> **username_5**: Is there anything about how statistics in psychology are taught differently in various places? I noticed this was at the univ. of Amsterdam. I did my BSc in the UK and we aren't really taught statistics, more how to use SPSS  
> **username_3**: @username_5 Anecdotally, I would say that sums up most undergrad statistics. (But don't get me started.) Also, given that many of the researchers tested were not trained at the UvA, only working there, I would say there's every reason to suspect that it's fairly representative.  

---

