  Vision Transformers (ViTs) have shown impressive performance and have become
a unified backbone for multiple vision tasks. However, both the attention
mechanism and multi-layer perceptrons (MLPs) in ViTs are not sufficiently
efficient due to dense multiplications, leading to costly training and
inference. To this end, we propose to reparameterize pre-trained ViTs with a
mixture of multiplication primitives, e.g., bitwise shifts and additions,
towards a new type of multiplication-reduced model, dubbed
$\textbf{ShiftAddViT}$, which aims to achieve end-to-end inference speedups on
GPUs without requiring training from scratch. Specifically, all
$\texttt{MatMuls}$ among queries, keys, and values are reparameterized using
additive kernels, after mapping queries and keys to binary codes in Hamming
space. The remaining MLPs or linear layers are then reparameterized with shift
kernels. We utilize TVM to implement and optimize those customized kernels for
practical hardware deployment on GPUs. We find that such a reparameterization
on attention maintains model accuracy, while inevitably leading to accuracy
drops when being applied to MLPs. To marry the best of both worlds, we further
propose a new mixture of experts (MoE) framework to reparameterize MLPs by
taking multiplication or its primitives as experts, e.g., multiplication and
shift, and designing a new latency-aware load-balancing loss. Such a loss helps
to train a generic router for assigning a dynamic amount of input tokens to
different experts according to their latency. Extensive experiments on various
2D/3D Transformer-based vision tasks consistently validate the effectiveness of
our proposed ShiftAddViT, achieving up to $\textbf{5.18$\times$}$ latency
reductions on GPUs and $\textbf{42.9}$% energy savings, while maintaining a
comparable accuracy as original or efficient ViTs.
