  A common technique in reinforcement learning is to evaluate the value
function from Monte Carlo simulations of a given policy, and use the estimated
value function to obtain a new policy which is greedy with respect to the
estimated value function. A well-known longstanding open problem in this
context is to prove the convergence of such a scheme when the value function of
a policy is estimated from data collected from a single sample path obtained
from implementing the policy (see page 99 of [Sutton and Barto, 2018], page 8
of [Tsitsiklis, 2002]). We present a solution to the open problem by showing
that a first-visit version of such a policy iteration scheme indeed converges
to the optimal policy provided that the policy improvement step uses lookahead
[Silver et al., 2016, Mnih et al., 2016, Silver et al., 2017b] rather than a
simple greedy policy improvement. We provide results both for the original open
problem in the tabular setting and also present extensions to the function
approximation setting, where we show that the policy resulting from the
algorithm performs close to the optimal policy within a function approximation
error.
