  We study the problem of off-policy evaluation (OPE) in Reinforcement Learning
(RL), where the aim is to estimate the performance of a new policy given
historical data that may have been generated by a different policy, or
policies. In particular, we introduce a novel doubly-robust estimator for the
OPE problem in RL, based on the Targeted Maximum Likelihood Estimation
principle from the statistical causal inference literature. We also introduce
several variance reduction techniques that lead to impressive performance gains
in off-policy evaluation. We show empirically that our estimator uniformly wins
over existing off-policy evaluation methods across multiple RL environments and
various levels of model misspecification. Finally, we further the existing
theoretical analysis of estimators for the RL off-policy estimation problem by
showing their $O_P(1/\sqrt{n})$ rate of convergence and characterizing their
asymptotic distribution.
