  We present Ordinary Differential Equation Variational Auto-Encoder
(ODE$^2$VAE), a latent second order ODE model for high-dimensional sequential
data. Leveraging the advances in deep generative models, ODE$^2$VAE can
simultaneously learn the embedding of high dimensional trajectories and infer
arbitrarily complex continuous-time latent dynamics. Our model explicitly
decomposes the latent space into momentum and position components and solves a
second order ODE system, which is in contrast to recurrent neural network (RNN)
based time series models and recently proposed black-box ODE techniques. In
order to account for uncertainty, we propose probabilistic latent ODE dynamics
parameterized by deep Bayesian neural networks. We demonstrate our approach on
motion capture, image rotation and bouncing balls datasets. We achieve
state-of-the-art performance in long term motion prediction and imputation
tasks.
