  This paper addresses the problem of model-free reinforcement learning for
Robust Markov Decision Process (RMDP) with large state spaces. The goal of the
RMDP framework is to find a policy that is robust against the parameter
uncertainties due to the mismatch between the simulator model and real-world
settings. We first propose the Robust Least Squares Policy Evaluation
algorithm, which is a multi-step online model-free learning algorithm for
policy evaluation. We prove the convergence of this algorithm using stochastic
approximation techniques. We then propose Robust Least Squares Policy Iteration
(RLSPI) algorithm for learning the optimal robust policy. We also give a
general weighted Euclidean norm bound on the error (closeness to optimality) of
the resulting policy. Finally, we demonstrate the performance of our RLSPI
algorithm on some standard benchmark problems.
