  Recently, a considerable number of studies in computer vision involves deep
neural architectures called vision transformers. Visual processing in these
models incorporates computational models that are claimed to implement
attention mechanisms. Despite an increasing body of work that attempts to
understand the role of attention mechanisms in vision transformers, their
effect is largely unknown. Here, we asked if the attention mechanisms in vision
transformers exhibit similar effects as those known in human visual attention.
To answer this question, we revisited the attention formulation in these models
and found that despite the name, computationally, these models perform a
special class of relaxation labeling with similarity grouping effects.
Additionally, whereas modern experimental findings reveal that human visual
attention involves both feed-forward and feedback mechanisms, the purely
feed-forward architecture of vision transformers suggests that attention in
these models will not have the same effects as those known in humans. To
quantify these observations, we evaluated grouping performance in a family of
vision transformers. Our results suggest that self-attention modules group
figures in the stimuli based on similarity in visual features such as color.
Also, in a singleton detection experiment as an instance of saliency detection,
we studied if these models exhibit similar effects as those of feed-forward
visual salience mechanisms utilized in human visual attention. We found that
generally, the transformer-based attention modules assign more salience either
to distractors or the ground. Together, our study suggests that the attention
mechanisms in vision transformers perform similarity grouping and not
attention.
