  We introduce a variational Bayesian neural network where the parameters are
governed via a probability distribution on random matrices. Specifically, we
employ a matrix variate Gaussian \cite{gupta1999matrix} parameter posterior
distribution where we explicitly model the covariance among the input and
output dimensions of each layer. Furthermore, with approximate covariance
matrices we can achieve a more efficient way to represent those correlations
that is also cheaper than fully factorized parameter posteriors. We further
show that with the "local reprarametrization trick"
\cite{kingma2015variational} on this posterior distribution we arrive at a
Gaussian Process \cite{rasmussen2006gaussian} interpretation of the hidden
units in each layer and we, similarly with \cite{gal2015dropout}, provide
connections with deep Gaussian processes. We continue in taking advantage of
this duality and incorporate "pseudo-data" \cite{snelson2005sparse} in our
model, which in turn allows for more efficient sampling while maintaining the
properties of the original model. The validity of the proposed approach is
verified through extensive experiments.
