  Recurrent neural networks (RNNs) have shown promising performance for
language modeling. However, traditional training of RNNs using back-propagation
through time often suffers from overfitting. One reason for this is that
stochastic optimization (used for large training sets) does not provide good
estimates of model uncertainty. This paper leverages recent advances in
stochastic gradient Markov Chain Monte Carlo (also appropriate for large
training sets) to learn weight uncertainty in RNNs. It yields a principled
Bayesian learning algorithm, adding gradient noise during training (enhancing
exploration of the model-parameter space) and model averaging when testing.
Extensive experiments on various RNN models and across a broad range of
applications demonstrate the superiority of the proposed approach over
stochastic optimization.
