  Transformer architecture has gained popularity due to its ability to scale
with large dataset. Consequently, there is a need to reduce the model size and
latency, especially for on-device deployment. We focus on vision transformer
proposed for image recognition task (Dosovitskiy et al., 2021), and explore the
application of different compression techniques such as low rank approximation
and pruning for this purpose. Specifically, we investigate a structured pruning
method proposed recently in Zhu et al. (2021) and find that mostly feedforward
blocks are pruned with this approach, that too, with severe degradation in
accuracy. We propose a hybrid compression approach to mitigate this where we
compress the attention blocks using low rank approximation and use the
previously mentioned pruning with a lower rate for feedforward blocks in each
transformer layer. Our technique results in 50% compression with 14% relative
increase in classification error whereas we obtain 44% compression with 20%
relative increase in error when only pruning is applied. We propose further
enhancements to bridge the accuracy gap but leave it as a future work.
