  In imitation learning, an agent learns how to behave in an environment with
an unknown cost function by mimicking expert demonstrations. Existing imitation
learning algorithms typically involve solving a sequence of planning or
reinforcement learning problems. Such algorithms are therefore not directly
applicable to large, high-dimensional environments, and their performance can
significantly degrade if the planning problems are not solved to optimality.
Under the apprenticeship learning formalism, we develop alternative model-free
algorithms for finding a parameterized stochastic policy that performs at least
as well as an expert policy on an unknown cost function, based on sample
trajectories from the expert. Our approach, based on policy gradients, scales
to large continuous environments with guaranteed convergence to local minima.
