  We introduce the first Neural Architecture Search (NAS) method to find a
better transformer architecture for image recognition. Recently, transformers
without CNN-based backbones are found to achieve impressive performance for
image recognition. However, the transformer is designed for NLP tasks and thus
could be sub-optimal when directly used for image recognition. In order to
improve the visual representation ability for transformers, we propose a new
search space and searching algorithm. Specifically, we introduce a locality
module that models the local correlations in images explicitly with fewer
computational cost. With the locality module, our search space is defined to
let the search algorithm freely trade off between global and local information
as well as optimizing the low-level design choice in each module. To tackle the
problem caused by huge search space, a hierarchical neural architecture search
method is proposed to search the optimal vision transformer from two levels
separately with the evolutionary algorithm. Extensive experiments on the
ImageNet dataset demonstrate that our method can find more discriminative and
efficient transformer variants than the ResNet family (e.g., ResNet101) and the
baseline ViT for image classification.
