  Stochastic recurrent neural networks with latent random variables of complex
dependency structures have shown to be more successful in modeling sequential
data than deterministic deep models. However, the majority of existing methods
have limited expressive power due to the Gaussian assumption of latent
variables. In this paper, we advocate learning implicit latent representations
using semi-implicit variational inference to further increase model
flexibility. Semi-implicit stochastic recurrent neural network(SIS-RNN) is
developed to enrich inferred model posteriors that may have no analytic density
functions, as long as independent random samples can be generated via
reparameterization. Extensive experiments in different tasks on real-world
datasets show that SIS-RNN outperforms the existing methods.
