  Recently, many reinforcement learning techniques were shown to have provable
guarantees in the simple case of linear dynamics, especially in problems like
linear quadratic regulators. However, in practice, many reinforcement learning
problems try to learn a policy directly from rich, high dimensional
representations such as images. Even if there is an underlying dynamics that is
linear in the correct latent representations (such as position and velocity),
the rich representation is likely to be nonlinear and can contain irrelevant
features. In this work we study a model where there is a hidden linear subspace
in which the dynamics is linear. For such a model we give an efficient
algorithm for extracting the linear subspace with linear dynamics. We then
extend our idea to extracting a nonlinear mapping, and empirically verify the
effectiveness of our approach in simple settings with rich observations.
