  Vision transformers have demonstrated remarkable success in a wide range of
computer vision tasks over the last years. However, their high computational
costs remain a significant barrier to their practical deployment. In
particular, the complexity of transformer models is quadratic with respect to
the number of input tokens. Therefore techniques that reduce the number of
input tokens that need to be processed have been proposed. This paper
introduces Learned Thresholds token Merging and Pruning (LTMP), a novel
approach that leverages the strengths of both token merging and token pruning.
LTMP uses learned threshold masking modules that dynamically determine which
tokens to merge and which to prune. We demonstrate our approach with extensive
experiments on vision transformers on the ImageNet classification task. Our
results demonstrate that LTMP achieves state-of-the-art accuracy across
reduction rates while requiring only a single fine-tuning epoch, which is an
order of magnitude faster than previous methods. Code is available at
https://github.com/Mxbonn/ltmp .
