  We introduce Embed to Control (E2C), a method for model learning and control
of non-linear dynamical systems from raw pixel images. E2C consists of a deep
generative model, belonging to the family of variational autoencoders, that
learns to generate image trajectories from a latent space in which the dynamics
is constrained to be locally linear. Our model is derived directly from an
optimal control formulation in latent space, supports long-term prediction of
image sequences and exhibits strong performance on a variety of complex control
problems.
