  We study the vision transformer structure in the mobile level in this paper,
and find a dramatic performance drop. We analyze the reason behind this
phenomenon, and propose a novel irregular patch embedding module and adaptive
patch fusion module to improve the performance. We conjecture that the vision
transformer blocks (which consist of multi-head attention and feed-forward
network) are more suitable to handle high-level information than low-level
features. The irregular patch embedding module extracts patches that contain
rich high-level information with different receptive fields. The transformer
blocks can obtain the most useful information from these irregular patches.
Then the processed patches pass the adaptive patch merging module to get the
final features for the classifier. With our proposed improvements, the
traditional uniform vision transformer structure can achieve state-of-the-art
results in mobile level. We improve the DeiT baseline by more than 9\% under
the mobile-level settings and surpass other transformer architectures like Swin
and CoaT by a large margin.
