  Many efforts have been devoted to training generative latent variable models
with autoregressive decoders, such as recurrent neural networks (RNN).
Stochastic recurrent models have been successful in capturing the variability
observed in natural sequential data such as speech. We unify successful ideas
from recently proposed architectures into a stochastic recurrent model: each
step in the sequence is associated with a latent variable that is used to
condition the recurrent dynamics for future steps. Training is performed with
amortized variational inference where the approximate posterior is augmented
with a RNN that runs backward through the sequence. In addition to maximizing
the variational lower bound, we ease training of the latent variables by adding
an auxiliary cost which forces them to reconstruct the state of the backward
recurrent network. This provides the latent variables with a task-independent
objective that enhances the performance of the overall model. We found this
strategy to perform better than alternative approaches such as KL annealing.
Although being conceptually simple, our model achieves state-of-the-art results
on standard speech benchmarks such as TIMIT and Blizzard and competitive
performance on sequential MNIST. Finally, we apply our model to language
modeling on the IMDB dataset where the auxiliary cost helps in learning
interpretable latent variables. Source Code:
\url{https://github.com/anirudh9119/zforcing_nips17}
