  Extensive research has demonstrated that deep neural networks (DNNs) are
prone to adversarial attacks. Although various defense mechanisms have been
proposed for image classification networks, fewer approaches exist for
video-based models that are used in security-sensitive applications like
surveillance. In this paper, we propose a novel yet simple algorithm called
Pseudo-Adversarial Training (PAT), to detect the adversarial frames in a video
without requiring knowledge of the attack. Our approach generates `transition
frames' that capture critical deviation from the original frames and eliminate
the components insignificant to the detection task. To avoid the necessity of
knowing the attack model, we produce `pseudo perturbations' to train our
detection network. Adversarial detection is then achieved through the use of
the detected frames. Experimental results on UCF-101 and 20BN-Jester datasets
show that PAT can detect the adversarial video frames and videos with a high
detection rate. We also unveil the potential reasons for the effectiveness of
the transition frames and pseudo perturbations through extensive experiments.
