  Bayesian neural networks (BNNs) augment deep networks with uncertainty
quantification by Bayesian treatment of the network weights. However, such
models face the challenge of Bayesian inference in a high-dimensional and
usually over-parameterized space. This paper investigates a new line of
Bayesian deep learning by performing Bayesian inference on network structure.
Instead of building structure from scratch inefficiently, we draw inspirations
from neural architecture search to represent the network structure. We then
develop an efficient stochastic variational inference approach which unifies
the learning of both network structure and weights. Empirically, our method
exhibits competitive predictive performance while preserving the benefits of
Bayesian principles across challenging scenarios. We also provide convincing
experimental justification for our modeling choice.
