  Pre-trained vision-language models (VLMs) have achieved impressive results in
a range of vision-language tasks. However, popular VLMs usually consist of
hundreds of millions of parameters which brings challenges for fine-tuning and
deployment in real-world applications due to space, memory, and latency
constraints. In this work, we introduce a distilling then pruning framework to
compress large vision-language models into smaller, faster, and more accurate
ones. We first shrink the size of a pre-trained large VLM and apply knowledge
distillation in the vision-language pre-training stage to obtain a
task-agnostic compact VLM. Then we propose a modal-adaptive pruning algorithm
to automatically infer the importance of vision and language modalities for
different downstream tasks and adaptively remove redundant structures and
neurons in different encoders with controllable target sparsity. We apply our
framework to train EfficientVLM, a fast and accurate vision-language model
consisting of 6 vision layers, 3 text layers, and 3 cross-modal fusion layers,
accounting for only 93 million parameters in total, which is 44.3% of the
teacher model. EfficientVLM retains 98.4% performance of the teacher model and
accelerates its inference speed by 2.2x. EfficientVLM achieves a large absolute
improvement over previous SoTA efficient VLMs of similar sizes by a large
margin on various vision-language tasks, including VQAv2 (+4.9%), NLVR2
(+5.6%), ITR (R@1 on TR +17.2%, on IR + 15.6% ) and COCO caption generation
(CIDEr +6.5), demonstrating a large potential on training lightweight VLMs.
