  Scatterplots are the most common way for statisticians, scientists, and the
public to visually detect relationships between measured variables. At the same
time, and despite widely publicized controversy, P-values remain the most
commonly used measure to statistically justify relationships identified between
variables. Here we measure the ability to detect statistically significant
relationships from scatterplots in a randomized trial of 2,039 students in a
statistics massive open online course (MOOC). Each subject was shown a random
set of scatterplots and asked to visually determine if the underlying
relationships were statistically significant at the P < 0.05 level. Subjects
correctly classified only 47.4% (95% CI: 45.1%-49.7%) of statistically
significant relationships, and 74.6% (95% CI: 72.5%-76.6%) of non-significant
relationships. Adding visual aids such as a best fit line or scatterplot smooth
increased the probability a relationship was called significant, regardless of
whether the relationship was actually significant. Classification of
statistically significant relationships improved on repeat attempts of the
survey, although classification of non-significant relationships did not. Our
results suggest: (1) that evidence-based data analysis can be used to identify
weaknesses in theoretical procedures in the hands of average users, (2) data
analysts can be trained to improve detection of statistically significant
results with practice, but (3) data analysts have incorrect intuition about
what statistically significant relationships look like, particularly for small
effects. We have built a web tool for people to compare scatterplots with their
corresponding p-values which is available here:
http://glimmer.rstudio.com/afisher/EDA/.
