  Vision transformers (ViTs) have achieved promising results on a variety of
Computer Vision tasks, however their quadratic complexity in the number of
input tokens has limited their application specially in resource-constrained
settings. Previous approaches that employ gradual token reduction to address
this challenge assume that token redundancy in one layer implies redundancy in
all the following layers. We empirically demonstrate that this assumption is
often not correct, i.e., tokens that are redundant in one layer can be useful
in later layers. We employ this key insight to propose a novel token
propagation controller (TPC) that incorporates two different
token-distributions, i.e., pause probability and restart probability to control
the reduction and reuse of tokens respectively, which results in more efficient
token utilization. To improve the estimates of token distributions, we propose
a smoothing mechanism that acts as a regularizer and helps remove noisy
outliers. Furthermore, to improve the training-stability of our proposed TPC,
we introduce a model stabilizer that is able to implicitly encode local image
structures and minimize accuracy fluctuations during model training. We present
extensive experimental results on the ImageNet-1K dataset using DeiT, LV-ViT
and Swin models to demonstrate the effectiveness of our proposed method. For
example, compared to baseline models, our proposed method improves the
inference speed of the DeiT-S by 250% while increasing the classification
accuracy by 1.0%.
