  The growing number of adversarial attacks in recent years gives attackers an
advantage over defenders, as defenders must train detectors after knowing the
types of attacks, and many models need to be maintained to ensure good
performance in detecting any upcoming attacks. We propose a way to end the
tug-of-war between attackers and defenders by treating adversarial attack
detection as an anomaly detection problem so that the detector is agnostic to
the attack. We quantify the statistical deviation caused by adversarial
perturbations in two aspects. The Least Significant Component Feature (LSCF)
quantifies the deviation of adversarial examples from the statistics of benign
samples and Hessian Feature (HF) reflects how adversarial examples distort the
landscape of the model's optima by measuring the local loss curvature.
Empirical results show that our method can achieve an overall ROC AUC of 94.9%,
89.7%, and 94.6% on CIFAR10, CIFAR100, and SVHN, respectively, and has
comparable performance to adversarial detectors trained with adversarial
examples on most of the attacks.
