  Transformer is a transformative framework that models sequential data and has
achieved remarkable performance on a wide range of tasks, but with high
computational and energy cost. To improve its efficiency, a popular choice is
to compress the models via binarization which constrains the floating-point
values into binary ones to save resource consumption owing to cheap bitwise
operations significantly. However, existing binarization methods only aim at
minimizing the information loss for the input distribution statistically, while
ignoring the pairwise similarity modeling at the core of the attention. To this
end, we propose a new binarization paradigm customized to high-dimensional
softmax attention via kernelized hashing, called EcoFormer, to map the original
queries and keys into low-dimensional binary codes in Hamming space. The
kernelized hash functions are learned to match the ground-truth similarity
relations extracted from the attention map in a self-supervised way. Based on
the equivalence between the inner product of binary codes and the Hamming
distance as well as the associative property of matrix multiplication, we can
approximate the attention in linear complexity by expressing it as a
dot-product of binary codes. Moreover, the compact binary representations of
queries and keys enable us to replace most of the expensive multiply-accumulate
operations in attention with simple accumulations to save considerable on-chip
energy footprint on edge devices. Extensive experiments on both vision and
language tasks show that EcoFormer consistently achieves comparable performance
with standard attentions while consuming much fewer resources. For example,
based on PVTv2-B0 and ImageNet-1K, Ecoformer achieves a 73% on-chip energy
footprint reduction with only a 0.33% performance drop compared to the standard
attention. Code is available at https://github.com/ziplab/EcoFormer.
