  Transformers are popular neural network models that use layers of
self-attention and fully-connected nodes with embedded tokens. Vision
Transformers (ViT) adapt transformers for image recognition tasks. In order to
do this, the images are split into patches and used as tokens. One issue with
ViT is the lack of inductive bias toward image structures. Because ViT was
adapted for image data from language modeling, the network does not explicitly
handle issues such as local translations, pixel information, and information
loss in the structures and features shared by multiple patches. Conversely,
Convolutional Neural Networks (CNN) incorporate this information. Thus, in this
paper, we propose the use of convolutional layers within ViT. Specifically, we
propose a model called a Vision Conformer (ViC) which replaces the Multi-Layer
Perceptron (MLP) in a ViT layer with a CNN. In addition, to use the CNN, we
proposed to reconstruct the image data after the self-attention in a reverse
embedding layer. Through the evaluation, we demonstrate that the proposed
convolutions help improve the classification ability of ViT.
