  Despite their compelling theoretical properties, Bayesian neural networks
(BNNs) tend to perform worse than frequentist methods in classification-based
uncertainty quantification (UQ) tasks such as out-of-distribution (OOD)
detection. In this paper, based on empirical findings in prior works, we
hypothesize that this issue is because even recent Bayesian methods have never
considered OOD data in their training processes, even though this "OOD
training" technique is an integral part of state-of-the-art frequentist UQ
methods. To validate this, we treat OOD data as a first-class citizen in BNN
training by exploring four different ways of incorporating OOD data into
Bayesian inference. We show in extensive experiments that OOD-trained BNNs are
competitive to recent frequentist baselines. This work thus provides strong
baselines for future work in Bayesian UQ.
