  We study reinforcement learning under model misspecification, where we do not
have access to the true environment but only to a reasonably close
approximation to it. We address this problem by extending the framework of
robust MDPs to the model-free Reinforcement Learning setting, where we do not
have access to the model parameters, but can only sample states from it. We
define robust versions of Q-learning, SARSA, and TD-learning and prove
convergence to an approximately optimal robust policy and approximate value
function respectively. We scale up the robust algorithms to large MDPs via
function approximation and prove convergence under two different settings. We
prove convergence of robust approximate policy iteration and robust approximate
value iteration for linear architectures (under mild assumptions). We also
define a robust loss function, the mean squared robust projected Bellman error
and give stochastic gradient descent algorithms that are guaranteed to converge
to a local minimum.
