  Vision Transformers (ViTs) have demonstrated state-of-the-art performance on
many Computer Vision Tasks. Unfortunately, deploying these large-scale ViTs is
resource-consuming and impossible for many mobile devices. While most in the
community are building for larger and larger ViTs, we ask a completely opposite
question: How small can a ViT be within the tradeoffs of accuracy and inference
latency that make it suitable for mobile deployment? We look into a few ViTs
specifically designed for mobile applications and observe that they modify the
transformer's architecture or are built around the combination of CNN and
transformer. Recent work has also attempted to create sparse ViT networks and
proposed alternatives to the attention module. In this paper, we study these
architectures, identify the challenges and analyze what really makes a vision
transformer suitable for mobile applications. We aim to serve as a baseline for
future research direction and hopefully lay the foundation to choose the
exemplary vision transformer architecture for your application running on
mobile devices.
