  Bayesian model comparison is often based on the posterior distribution over
the set of compared models. This distribution is often observed to concentrate
on a single model even when other measures of model fit or forecasting ability
indicate no strong preference. Furthermore, a moderate change in the data
sample can easily shift the posterior model probabilities to concentrate on
another model. We document overconfidence in two high-profile applications in
economics and neuroscience. To shed more light on the sources of overconfidence
we derive the sampling variance of the Bayes factor in univariate and
multivariate linear regression. The results show that overconfidence is likely
to happen when i) the compared models give very different approximations of the
data-generating process, ii) the models are very flexible with large degrees of
freedom that are not shared between the models, and iii) the models
underestimate the true variability in the data.
