  Many animals and humans process the visual field with a varying spatial
resolution (foveated vision) and use peripheral processing to make eye
movements and point the fovea to acquire high-resolution information about
objects of interest. This architecture results in computationally efficient
rapid scene exploration. Recent progress in self-attention-based Vision
Transformers, an alternative to the traditionally convolution-reliant computer
vision systems. However, the Transformer models do not explicitly model the
foveated properties of the visual system nor the interaction between eye
movements and the classification task. We propose Foveated Transformer
(FoveaTer) model, which uses pooling regions and eye movements to perform
object classification tasks using a Vision Transformer architecture. Using
square pooling regions or biologically-inspired radial-polar pooling regions,
our proposed model pools the image features from the convolution backbone and
uses the pooled features as an input to transformer layers. It decides on
subsequent fixation location based on the attention assigned by the Transformer
to various locations from past and present fixations. It dynamically allocates
more fixation/computational resources to more challenging images before making
the final image category decision. Using five ablation studies, we evaluate the
contribution of different components of the Foveated model. We perform a
psychophysics scene categorization task and use the experimental data to find a
suitable radial-polar pooling region combination. We also show that the
Foveated model better explains the human decisions in a scene categorization
task than a Baseline model. We demonstrate our model's robustness against PGD
adversarial attacks with both types of pooling regions, where we see the
Foveated model outperform the Baseline model.
