  This paper uses symmetry to make Convolutional Neural Network classifiers
(CNNs) robust against adversarial perturbation attacks. Such attacks add
perturbation to original images to generate adversarial images that fool
classifiers such as road sign classifiers of autonomous vehicles. Although
symmetry is a pervasive aspect of the natural world, CNNs are unable to handle
symmetry well. For example, a CNN can classify an image differently from its
mirror image. For an adversarial image that misclassifies with a wrong label
$l_w$, CNN inability to handle symmetry means that a symmetric adversarial
image can classify differently from the wrong label $l_w$. Further than that,
we find that the classification of a symmetric adversarial image reverts to the
correct label. To classify an image when adversaries are unaware of the
defense, we apply symmetry to the image and use the classification label of the
symmetric image. To classify an image when adversaries are aware of the
defense, we use mirror symmetry and pixel inversion symmetry to form a symmetry
group. We apply all the group symmetries to the image and decide on the output
label based on the agreement of any two of the classification labels of the
symmetry images. Adaptive attacks fail because they need to rely on loss
functions that use conflicting CNN output values for symmetric images. Without
attack knowledge, the proposed symmetry defense succeeds against both
gradient-based and random-search attacks, with up to near-default accuracies
for ImageNet. The defense even improves the classification accuracy of original
images.
