  Robust reinforcement learning (RL) is to find a policy that optimizes the
worst-case performance over an uncertainty set of MDPs. In this paper, we focus
on model-free robust RL, where the uncertainty set is defined to be centering
at a misspecified MDP that generates a single sample trajectory sequentially
and is assumed to be unknown. We develop a sample-based approach to estimate
the unknown uncertainty set and design a robust Q-learning algorithm (tabular
case) and robust TDC algorithm (function approximation setting), which can be
implemented in an online and incremental fashion. For the robust Q-learning
algorithm, we prove that it converges to the optimal robust Q function, and for
the robust TDC algorithm, we prove that it converges asymptotically to some
stationary points. Unlike the results in [Roy et al., 2017], our algorithms do
not need any additional conditions on the discount factor to guarantee the
convergence. We further characterize the finite-time error bounds of the two
algorithms and show that both the robust Q-learning and robust TDC algorithms
converge as fast as their vanilla counterparts(within a constant factor). Our
numerical experiments further demonstrate the robustness of our algorithms. Our
approach can be readily extended to robustify many other algorithms, e.g., TD,
SARSA, and other GTD algorithms.
