  Abnormal states in deep reinforcement learning~(RL) are states that are
beyond the scope of an RL policy. Such states may lead to sub-optimal and
unsafe decision making for the RL system, impeding its deployment in real
scenarios. In this paper, we propose a simple yet effective anomaly detection
framework for deep RL algorithms that simultaneously considers random,
adversarial and out-of-distribution~(OOD) state outliers. In particular, we
attain the class-conditional distributions for each action class under the
Gaussian assumption, and rely on these distributions to discriminate between
inliers and outliers based on Mahalanobis Distance~(MD) and Robust Mahalanobis
Distance. We conduct extensive experiments on Atari games that verify the
effectiveness of our detection strategies. To the best of our knowledge, we
present the first in-detail study of statistical and adversarial anomaly
detection in deep RL algorithms. This simple unified anomaly detection paves
the way towards deploying safe RL systems in real-world applications.
