  Recently, self-attention (SA) structures became popular in computer vision
fields. They have locally independent filters and can use large kernels, which
contradicts the previously popular convolutional neural networks (CNNs). CNNs
success was attributed to the hard-coded inductive biases of locality and
spatial invariance. However, recent studies have shown that inductive biases in
CNNs are too restrictive. On the other hand, the relative position encodings,
similar to depthwise (DW) convolution, are necessary for the local SA networks,
which indicates that the SA structures are not entirely spatially variant.
Hence, we would like to determine which part of inductive biases contributes to
the success of the local SA structures. To do so, we introduced context-aware
decomposed attention (CADA), which decomposes attention maps into multiple
trainable base kernels and accumulates them using context-aware (CA)
parameters. This way, we could identify the link between the CNNs and SA
networks. We conducted ablation studies using the ResNet50 applied to the
ImageNet classification task. DW convolution could have a large locality
without increasing computational costs compared to CNNs, but the accuracy
saturates with larger kernels. CADA follows this characteristic of locality. We
showed that context awareness was the crucial property; however, large local
information was not necessary to construct CA parameters. Even though no
spatial invariance makes training difficult, more relaxed spatial invariance
gave better accuracy than strict spatial invariance. Also, additional strong
spatial invariance through relative position encoding was preferable. We
extended these experiments to filters for downsampling and showed that locality
bias is more critical for downsampling but can remove the strong locality bias
using relaxed spatial invariance.
