  In research on eye movements in reading, it is common to analyze a number of
canonical dependent measures to study how the effects of a manipulation unfold
over time. Although this gives rise to the well-known multiple comparisons
problem, i.e. an inflated probability that the null hypothesis is incorrectly
rejected (Type I error), it is accepted standard practice not to apply any
correction procedures. Instead, there appears to be a widespread belief that
corrections are not necessary because the increase in false positives is too
small to matter. To our knowledge, no formal argument has ever been presented
to justify this assumption. Here, we report a computational investigation of
this issue using Monte Carlo simulations. Our results show that, contrary to
conventional wisdom, false positives are increased to unacceptable levels when
no corrections are applied. Our simulations also show that counter-measures
like the Bonferroni correction keep false positives in check while reducing
statistical power only moderately. Hence, there is little reason why such
corrections should not be made a standard requirement. Further, we discuss
three statistical illusions that can arise when statistical power is low, and
we show how power can be improved to prevent these illusions. In sum, our work
renders a detailed picture of the various types of statistical errors than can
occur in studies of reading behavior and we provide concrete guidance about how
these errors can be avoided.
