  Imitation learning (IL) aims to mimic the behavior of an expert in a
sequential decision making task by learning from demonstrations, and has been
widely applied to robotics, autonomous driving, and autoregressive text
generation. The simplest approach to IL, behavior cloning (BC), is thought to
incur sample complexity with unfavorable quadratic dependence on the problem
horizon, motivating a variety of different online algorithms that attain
improved linear horizon dependence under stronger assumptions on the data and
the learner's access to the expert.
  We revisit the apparent gap between offline and online IL from a
learning-theoretic perspective, with a focus on general policy classes up to
and including deep neural networks. Through a new analysis of behavior cloning
with the logarithmic loss, we show that it is possible to achieve
horizon-independent sample complexity in offline IL whenever (i) the range of
the cumulative payoffs is controlled, and (ii) an appropriate notion of
supervised learning complexity for the policy class is controlled. Specializing
our results to deterministic, stationary policies, we show that the gap between
offline and online IL is not fundamental: (i) it is possible to achieve linear
dependence on horizon in offline IL under dense rewards (matching what was
previously only known to be achievable in online IL); and (ii) without further
assumptions on the policy class, online IL cannot improve over offline IL with
the logarithmic loss, even in benign MDPs. We complement our theoretical
results with experiments on standard RL tasks and autoregressive language
generation to validate the practical relevance of our findings.
