  Leveraging advances in variational inference, we propose to enhance recurrent
neural networks with latent variables, resulting in Stochastic Recurrent
Networks (STORNs). The model i) can be trained with stochastic gradient
methods, ii) allows structured and multi-modal conditionals at each time step,
iii) features a reliable estimator of the marginal likelihood and iv) is a
generalisation of deterministic recurrent neural networks. We evaluate the
method on four polyphonic musical data sets and motion capture data.
