  We present a new algorithm for imitation learning in infinite horizon linear
MDPs dubbed ILARL which greatly improves the bound on the number of
trajectories that the learner needs to sample from the environment. In
particular, we remove exploration assumptions required in previous works and we
improve the dependence on the desired accuracy $\epsilon$ from
$\mathcal{O}\br{\epsilon^{-5}}$ to $\mathcal{O}\br{\epsilon^{-4}}$. Our result
relies on a connection between imitation learning and online learning in MDPs
with adversarial losses. For the latter setting, we present the first result
for infinite horizon linear MDP which may be of independent interest. Moreover,
we are able to provide a strengthen result for the finite horizon case where we
achieve $\mathcal{O}\br{\epsilon^{-2}}$. Numerical experiments with linear
function approximation shows that ILARL outperforms other commonly used
algorithms.
