  Deep learning models have been shown to be vulnerable to adversarial attacks.
In particular, gradient-based attacks have demonstrated high success rates
recently. The gradient measures how each image pixel affects the model output,
which contains critical information for generating malicious perturbations. In
this paper, we show that the gradients can also be exploited as a powerful
weapon to defend against adversarial attacks. By using both gradient maps and
adversarial images as inputs, we propose a Two-stream Restoration Network (TRN)
to restore the adversarial images. To optimally restore the perturbed images
with two streams of inputs, a Gradient Map Estimation Mechanism is proposed to
estimate the gradients of adversarial images, and a Fusion Block is designed in
TRN to explore and fuse the information in two streams. Once trained, our TRN
can defend against a wide range of attack methods without significantly
degrading the performance of benign inputs. Also, our method is generalizable,
scalable, and hard to bypass. Experimental results on CIFAR10, SVHN, and
Fashion MNIST demonstrate that our method outperforms state-of-the-art defense
methods.
