  Neural networks with self-attention (a.k.a. Transformers) like ViT and Swin
have emerged as a better alternative to traditional convolutional neural
networks (CNNs). However, our understanding of how the new architecture works
is still limited. In this paper, we focus on the phenomenon that Transformers
show higher robustness against corruptions than CNNs, while not being
overconfident. This is contrary to the intuition that robustness increases with
confidence. We resolve this contradiction by empirically investigating how the
output of the penultimate layer moves in the representation space as the input
data moves linearly within a small area. In particular, we show the following.
(1) While CNNs exhibit fairly linear relationship between the input and output
movements, Transformers show nonlinear relationship for some data. For those
data, the output of Transformers moves in a curved trajectory as the input
moves linearly. (2) When a data is located in a curved region, it is hard to
move it out of the decision region since the output moves along a curved
trajectory instead of a straight line to the decision boundary, resulting in
high robustness of Transformers. (3) If a data is slightly modified to jump out
of the curved region, the movements afterwards become linear and the output
goes to the decision boundary directly. In other words, there does exist a
decision boundary near the data, which is hard to find only because of the
curved representation space. This explains the underconfident prediction of
Transformers. Also, we examine mathematical properties of the attention
operation that induce nonlinear response to linear perturbation. Finally, we
share our additional findings, regarding what contributes to the curved
representation space of Transformers, and how the curvedness evolves during
training.
