  We consider imitation learning problems where the learner's ability to mimic
the expert increases throughout the course of an episode as more information is
revealed. One example of this is when the expert has access to privileged
information: while the learner might not be able to accurately reproduce expert
behavior early on in an episode, by considering the entire history of states
and actions, they might be able to eventually identify the hidden context and
act as the expert would. We prove that on-policy imitation learning algorithms
(with or without access to a queryable expert) are better equipped to handle
these sorts of asymptotically realizable problems than off-policy methods. This
is because on-policy algorithms provably learn to recover from their initially
suboptimal actions, while off-policy methods treat their suboptimal past
actions as though they came from the expert. This often manifests as a latching
behavior: a naive repetition of past actions. We conduct experiments in a toy
bandit domain that show that there exist sharp phase transitions of whether
off-policy approaches are able to match expert performance asymptotically, in
contrast to the uniformly good performance of on-policy approaches. We
demonstrate that on several continuous control tasks, on-policy approaches are
able to use history to identify the context while off-policy approaches
actually perform worse when given access to history.
