  We propose $\textit{iterative inversion}$ -- an algorithm for learning an
inverse function without input-output pairs, but only with samples from the
desired output distribution and access to the forward function. The key
challenge is a $\textit{distribution shift}$ between the desired outputs and
the outputs of an initial random guess, and we prove that iterative inversion
can steer the learning correctly, under rather strict conditions on the
function. We apply iterative inversion to learn control. Our input is a set of
demonstrations of desired behavior, given as video embeddings of trajectories
(without actions), and our method iteratively learns to imitate trajectories
generated by the current policy, perturbed by random exploration noise. Our
approach does not require rewards, and only employs supervised learning, which
can be easily scaled to use state-of-the-art trajectory embedding techniques
and policy representations. Indeed, with a VQ-VAE embedding, and a
transformer-based policy, we demonstrate non-trivial continuous control on
several tasks. Further, we report an improved performance on imitating diverse
behaviors compared to reward based methods.
