  We propose an approach for improving sequence modeling based on
autoregressive normalizing flows. Each autoregressive transform, acting across
time, serves as a moving frame of reference, removing temporal correlations,
and simplifying the modeling of higher-level dynamics. This technique provides
a simple, general-purpose method for improving sequence modeling, with
connections to existing and classical techniques. We demonstrate the proposed
approach both with standalone flow-based models and as a component within
sequential latent variable models. Results are presented on three benchmark
video datasets, where autoregressive flow-based dynamics improve log-likelihood
performance over baseline models. Finally, we illustrate the decorrelation and
improved generalization properties of using flow-based dynamics.
