  Vision Transformers (ViTs) have emerged as state-of-the-art models for
various vision tasks recently. However, their heavy computation costs remain
daunting for resource-limited devices. Consequently, researchers have dedicated
themselves to compressing redundant information in ViTs for acceleration.
However, they generally sparsely drop redundant image tokens by token pruning
or brutally remove channels by channel pruning, leading to a sub-optimal
balance between model performance and inference speed. They are also
disadvantageous in transferring compressed models to downstream vision tasks
that require the spatial structure of images, such as semantic segmentation. To
tackle these issues, we propose a joint compression method for ViTs that offers
both high accuracy and fast inference speed, while also maintaining favorable
transferability to downstream tasks (CAIT). Specifically, we introduce an
asymmetric token merging (ATME) strategy to effectively integrate neighboring
tokens. It can successfully compress redundant token information while
preserving the spatial structure of images. We further employ a consistent
dynamic channel pruning (CDCP) strategy to dynamically prune unimportant
channels in ViTs. Thanks to CDCP, insignificant channels in multi-head
self-attention modules of ViTs can be pruned uniformly, greatly enhancing the
model compression. Extensive experiments on benchmark datasets demonstrate that
our proposed method can achieve state-of-the-art performance across various
ViTs. For example, our pruned DeiT-Tiny and DeiT-Small achieve speedups of
1.7$\times$ and 1.9$\times$, respectively, without accuracy drops on ImageNet.
On the ADE20k segmentation dataset, our method can enjoy up to 1.31$\times$
speedups with comparable mIoU. Our code will be publicly available.
