  Due to the high price and heavy energy consumption of GPUs, deploying deep
models on IoT devices such as microcontrollers makes significant contributions
for ecological AI. Conventional methods successfully enable convolutional
neural network inference of high resolution images on microcontrollers, while
the framework for vision transformers that achieve the state-of-the-art
performance in many vision applications still remains unexplored. In this
paper, we propose a hardware-algorithm co-optimizations method called MCUFormer
to deploy vision transformers on microcontrollers with extremely limited
memory, where we jointly design transformer architecture and construct the
inference operator library to fit the memory resource constraint. More
specifically, we generalize the one-shot network architecture search (NAS) to
discover the optimal architecture with highest task performance given the
memory budget from the microcontrollers, where we enlarge the existing search
space of vision transformers by considering the low-rank decomposition
dimensions and patch resolution for memory reduction. For the construction of
the inference operator library of vision transformers, we schedule the memory
buffer during inference through operator integration, patch embedding
decomposition, and token overwriting, allowing the memory buffer to be fully
utilized to adapt to the forward pass of the vision transformer. Experimental
results demonstrate that our MCUFormer achieves 73.62\% top-1 accuracy on
ImageNet for image classification with 320KB memory on STM32F746
microcontroller. Code is available at https://github.com/liangyn22/MCUFormer.
