  Humans see low and high spatial frequency components at the same time, and
combine the information from both to form a visual scene. Drawing on this
neuroscientific inspiration, we propose an altered Vision Transformer
architecture where patches from scaled down versions of the input image are
added to the input of the first Transformer Encoder layer. We name this model
Retina Vision Transformer (RetinaViT) due to its inspiration from the human
visual system. Our experiments show that when trained on the ImageNet-1K
dataset with a moderate configuration, RetinaViT achieves a 3.3% performance
improvement over the original ViT. We hypothesize that this improvement can be
attributed to the inclusion of low spatial frequency components in the input,
which improves the ability to capture structural features, and to select and
forward important features to deeper layers. RetinaViT thereby opens doors to
further investigations into vertical pathways and attention patterns.
