  Transformers have been recently adapted for large scale image classification,
achieving high scores shaking up the long supremacy of convolutional neural
networks. However the optimization of image transformers has been little
studied so far. In this work, we build and optimize deeper transformer networks
for image classification. In particular, we investigate the interplay of
architecture and optimization of such dedicated transformers. We make two
transformers architecture changes that significantly improve the accuracy of
deep transformers. This leads us to produce models whose performance does not
saturate early with more depth, for instance we obtain 86.5% top-1 accuracy on
Imagenet when training with no external data, we thus attain the current SOTA
with less FLOPs and parameters. Moreover, our best model establishes the new
state of the art on Imagenet with Reassessed labels and Imagenet-V2 / match
frequency, in the setting with no additional training data. We share our code
and models.
