  Bayesian linear mixed-effects models and Bayesian ANOVA are increasingly
being used in the cognitive sciences to perform null hypothesis tests, where a
null hypothesis that an effect is zero is compared with an alternative
hypothesis that the effect exists and is different from zero. While software
tools for Bayes factor null hypothesis tests are easily accessible, how to
specify the data and the model correctly is often not clear. In Bayesian
approaches, many authors use data aggregation at the by-subject level and
estimate Bayes factors on aggregated data. Here, we use simulation-based
calibration for model inference applied to several example experimental designs
to demonstrate that, as with frequentist analysis, such null hypothesis tests
on aggregated data can be problematic in Bayesian analysis. Specifically, when
random slope variances differ (i.e., violated sphericity assumption), Bayes
factors are too conservative for contrasts where the variance is small and they
are too liberal for contrasts where the variance is large. Running Bayesian
ANOVA on aggregated data can - if the sphericity assumption is violated -
likewise lead to biased Bayes factor results. Moreover, Bayes factors for
by-subject aggregated data are biased (too liberal) when random item slope
variance is present but ignored in the analysis. These problems can be
circumvented or reduced by running Bayesian linear mixed-effects models on
non-aggregated data such as on individual trials, and by explicitly modeling
the full random effects structure. Reproducible code is available from
\url{https://osf.io/mjf47/}.
