  We develop two Regression Monte Carlo algorithms (value and performance
iteration) to solve general problems of optimal stochastic control of
discrete-time Markov processes. We formulate our method within an innovative
framework that allow us to prove the speed of convergence of our numerical
schemes. We rely on the Regress Later approach unlike other attempts which
employ the Regress Now technique. We exploit error bounds obtained in our
proofs, along with numerical experiments, to investigate differences between
the value and performance iteration approaches. Introduced in Tsitsiklis and
VanRoy [2001] and Longstaff and Schwartz [2001] respectively, their
characteristics have gone largely unnoticed in the literature; we show however
that their differences are paramount in practical solution of stochastic
control problems. Finally, we provide some guidelines for the tuning of our
algorithms.
