  Data-agnostic quasi-imperceptible perturbations on inputs are known to
degrade recognition accuracy of deep convolutional networks severely. This
phenomenon is considered to be a potential security issue. Moreover, some
results on statistical generalization guarantees indicate that the phenomenon
can be a key to improve the networks' generalization. However, the
characteristics of the shared directions of such harmful perturbations remain
unknown. Our primal finding is that convolutional networks are sensitive to the
directions of Fourier basis functions. We derived the property by specializing
a hypothesis of the cause of the sensitivity, known as the linearity of neural
networks, to convolutional networks and empirically validated it. As a
by-product of the analysis, we propose an algorithm to create shift-invariant
universal adversarial perturbations available in black-box settings.
