  Vision Transformers (ViT) have recently demonstrated success across a myriad
of computer vision tasks. However, their elevated computational demands pose
significant challenges for real-world deployment. While low-rank approximation
stands out as a renowned method to reduce computational loads, efficiently
automating the target rank selection in ViT remains a challenge. Drawing from
the notable similarity and alignment between the processes of rank selection
and One-Shot NAS, we introduce FLORA, an end-to-end automatic framework based
on NAS. To overcome the design challenge of supernet posed by vast search
space, FLORA employs a low-rank aware candidate filtering strategy. This method
adeptly identifies and eliminates underperforming candidates, effectively
alleviating potential undertraining and interference among subnetworks. To
further enhance the quality of low-rank supernets, we design a low-rank
specific training paradigm. First, we propose weight inheritance to construct
supernet and enable gradient sharing among low-rank modules. Secondly, we adopt
low-rank aware sampling to strategically allocate training resources, taking
into account inherited information from pre-trained models. Empirical results
underscore FLORA's efficacy. With our method, a more fine-grained rank
configuration can be generated automatically and yield up to 33% extra FLOPs
reduction compared to a simple uniform configuration. More specific,
FLORA-DeiT-B/FLORA-Swin-B can save up to 55%/42% FLOPs almost without
performance degradtion. Importantly, FLORA boasts both versatility and
orthogonality, offering an extra 21%-26% FLOPs reduction when integrated with
leading compression techniques or compact hybrid structures. Our code is
publicly available at https://github.com/shadowpa0327/FLORA.
