  Deploying reinforcement learning (RL) systems requires robustness to
uncertainty and model misspecification, yet prior robust RL methods typically
only study noise introduced independently across time. However, practical
sources of uncertainty are usually coupled across time. We formally introduce
temporally-coupled perturbations, presenting a novel challenge for existing
robust RL methods. To tackle this challenge, we propose GRAD, a novel
game-theoretic approach that treats the temporally-coupled robust RL problem as
a partially observable two-player zero-sum game. By finding an approximate
equilibrium within this game, GRAD optimizes for general robustness against
temporally-coupled perturbations. Experiments on continuous control tasks
demonstrate that, compared with prior methods, our approach achieves a higher
degree of robustness to various types of attacks on different attack domains,
both in settings with temporally-coupled perturbations and decoupled
perturbations.
