  We introduce the variational filtering EM algorithm, a simple,
general-purpose method for performing variational inference in dynamical latent
variable models using information from only past and present variables, i.e.
filtering. The algorithm is derived from the variational objective in the
filtering setting and consists of an optimization procedure at each time step.
By performing each inference optimization procedure with an iterative amortized
inference model, we obtain a computationally efficient implementation of the
algorithm, which we call amortized variational filtering. We present
experiments demonstrating that this general-purpose method improves performance
across several deep dynamical latent variable models.
