  In this article a novel approach for training deep neural networks using
Bayesian techniques is presented. The Bayesian methodology allows for an easy
evaluation of model uncertainty and additionally is robust to overfitting.
These are commonly the two main problems classical, i.e. non-Bayesian,
architectures have to struggle with. The proposed approach applies variational
inference in order to approximate the intractable posterior distribution. In
particular, the variational distribution is defined as product of multiple
multivariate normal distributions with tridiagonal covariance matrices. Each
single normal distribution belongs either to the weights, or to the biases
corresponding to one network layer. The layer-wise a posteriori variances are
defined based on the corresponding expectation values and further the
correlations are assumed to be identical. Therefore, only a few additional
parameters need to be optimized compared to non-Bayesian settings. The novel
approach is successfully evaluated on basis of the popular benchmark datasets
MNIST and CIFAR-10.
