  Vision transformers using self-attention or its proposed alternatives have
demonstrated promising results in many image related tasks. However, the
underpinning inductive bias of attention is not well understood. To address
this issue, this paper analyzes attention through the lens of convex duality.
For the non-linear dot-product self-attention, and alternative mechanisms such
as MLP-mixer and Fourier Neural Operator (FNO), we derive equivalent
finite-dimensional convex problems that are interpretable and solvable to
global optimality. The convex programs lead to {\it block nuclear-norm
regularization} that promotes low rank in the latent feature and token
dimensions. In particular, we show how self-attention networks implicitly
clusters the tokens, based on their latent similarity. We conduct experiments
for transferring a pre-trained transformer backbone for CIFAR-100
classification by fine-tuning a variety of convex attention heads. The results
indicate the merits of the bias induced by attention compared with the existing
MLP or linear heads.
