  Deep Reinforcement Learning (DRL) policies are critically vulnerable to
adversarial noise in observations, posing severe risks in safety-critical
scenarios. For example, a self-driving car receiving manipulated sensory inputs
about traffic signs could lead to catastrophic outcomes. Existing strategies to
fortify RL algorithms against such adversarial perturbations generally fall
into two categories: (a) using regularization methods that enhance robustness
by incorporating adversarial loss terms into the value objectives, and (b)
adopting "maximin" principles, which focus on maximizing the minimum value to
ensure robustness. While regularization methods reduce the likelihood of
successful attacks, their effectiveness drops significantly if an attack does
succeed. On the other hand, maximin objectives, although robust, tend to be
overly conservative. To address this challenge, we introduce a novel objective
called Adversarial Counterfactual Error (ACoE), which naturally balances
optimizing value and robustness against adversarial attacks. To optimize ACoE
in a scalable manner in model-free settings, we propose a theoretically
justified surrogate objective known as Cumulative-ACoE (C-ACoE). The core idea
of optimizing C-ACoE is utilizing the belief about the underlying true state
given the adversarially perturbed observation. Our empirical evaluations
demonstrate that our method outperforms current state-of-the-art approaches for
addressing adversarial RL problems across all established benchmarks (MuJoCo,
Atari, and Highway) used in the literature.
