  While real-world applications of reinforcement learning are becoming popular,
the security and robustness of RL systems are worthy of more attention and
exploration. In particular, recent works have revealed that, in a multi-agent
RL environment, backdoor trigger actions can be injected into a victim agent
(a.k.a. Trojan agent), which can result in a catastrophic failure as soon as it
sees the backdoor trigger action. To ensure the security of RL agents against
malicious backdoors, in this work, we propose the problem of Backdoor Detection
in a multi-agent competitive reinforcement learning system, with the objective
of detecting Trojan agents as well as the corresponding potential trigger
actions, and further trying to mitigate their Trojan behavior. In order to
solve this problem, we propose PolicyCleanse that is based on the property that
the activated Trojan agents accumulated rewards degrade noticeably after
several timesteps. Along with PolicyCleanse, we also design a machine
unlearning-based approach that can effectively mitigate the detected backdoor.
Extensive experiments demonstrate that the proposed methods can accurately
detect Trojan agents, and outperform existing backdoor mitigation baseline
approaches by at least 3% in winning rate across various types of agents and
environments.
