  In this work we explore a straightforward variational Bayes scheme for
Recurrent Neural Networks. Firstly, we show that a simple adaptation of
truncated backpropagation through time can yield good quality uncertainty
estimates and superior regularisation at only a small extra computational cost
during training, also reducing the amount of parameters by 80\%. Secondly, we
demonstrate how a novel kind of posterior approximation yields further
improvements to the performance of Bayesian RNNs. We incorporate local gradient
information into the approximate posterior to sharpen it around the current
batch statistics. We show how this technique is not exclusive to recurrent
neural networks and can be applied more widely to train Bayesian neural
networks. We also empirically demonstrate how Bayesian RNNs are superior to
traditional RNNs on a language modelling benchmark and an image captioning
task, as well as showing how each of these methods improve our model over a
variety of other schemes for training them. We also introduce a new benchmark
for studying uncertainty for language models so future methods can be easily
compared.
