  In Natural Language Processing (NLP), Transformers have already
revolutionized the field by utilizing an attention-based encoder-decoder model.
Recently, some pioneering works have employed Transformer-like architectures in
Computer Vision (CV) and they have reported outstanding performance of these
architectures in tasks such as image classification, object detection, and
semantic segmentation. Vision Transformers (ViTs) have demonstrated impressive
performance improvements over Convolutional Neural Networks (CNNs) due to their
competitive modelling capabilities. However, these architectures demand massive
computational resources which makes these models difficult to be deployed in
the resource-constrained applications. Many solutions have been developed to
combat this issue, such as compressive transformers and compression functions
such as dilated convolution, min-max pooling, 1D convolution, etc. Model
compression has recently attracted considerable research attention as a
potential remedy. A number of model compression methods have been proposed in
the literature such as weight quantization, weight multiplexing, pruning and
Knowledge Distillation (KD). However, techniques like weight quantization,
pruning and weight multiplexing typically involve complex pipelines for
performing the compression. KD has been found to be a simple and much effective
model compression technique that allows a relatively simple model to perform
tasks almost as accurately as a complex model. This paper discusses various
approaches based upon KD for effective compression of ViT models. The paper
elucidates the role played by KD in reducing the computational and memory
requirements of these models. The paper also presents the various challenges
faced by ViTs that are yet to be resolved.
