  Bayesian neural networks (BNNs) have recently regained a significant amount
of attention in the deep learning community due to the development of scalable
approximate Bayesian inference techniques. There are several advantages of
using a Bayesian approach: Parameter and prediction uncertainties become easily
available, facilitating rigorous statistical analysis. Furthermore, prior
knowledge can be incorporated. However, so far, there have been no scalable
techniques capable of combining both structural and parameter uncertainty. In
this paper, we apply the concept of model uncertainty as a framework for
structural learning in BNNs and hence make inference in the joint space of
structures/models and parameters. Moreover, we suggest an adaptation of a
scalable variational inference approach with reparametrization of marginal
inclusion probabilities to incorporate the model space constraints.
Experimental results on a range of benchmark datasets show that we obtain
comparable accuracy results with the competing models, but based on methods
that are much more sparse than ordinary BNNs.
