  Bayes factor null hypothesis tests provide a viable alternative to
frequentist measures of evidence quantification. Bayes factors for realistic
interesting models cannot be calculated exactly, but have to be estimated,
which involves approximations to complex integrals. Crucially, the accuracy of
these estimates, i.e., whether an estimated Bayes factor corresponds to the
true Bayes factor, is unknown, and may depend on data, prior, and likelihood.
We have recently developed a novel statistical procedure, namely
simulation-based calibration (SBC) for Bayes factors, to test for a given
analysis, whether the computed Bayes factors are accurate. Here, we use SBC for
Bayes factors to test for some common cognitive designs, whether Bayes factors
are estimated accurately. We use the bridgesampling/brms packages as well as
the BayesFactor package in R. We find that Bayes factor estimates are accurate
and exhibit only little bias in Latin square designs with (a) random effects
for subjects only and (b) for crossed random effects for subjects and items,
but a single fixed-factor. However, Bayes factor estimates turn out biased and
liberal in a 2x2 design with crossed random effects for subjects and items.
These results suggest that researchers should test for their individual
analysis, whether Bayes factor estimates are accurate. Moreover, future
research is needed to determine the boundary conditions under which Bayes
factor estimates are accurate or biased, as well as software development to
improve estimation accuracy.
