  This paper studies the curious phenomenon for machine learning models with
Transformer architectures that their activation maps are sparse. By activation
map we refer to the intermediate output of the multi-layer perceptrons (MLPs)
after a ReLU activation function, and by sparse we mean that on average very
few entries (e.g., 3.0% for T5-Base and 6.3% for ViT-B16) are nonzero for each
input to MLP. Moreover, larger Transformers with more layers and wider MLP
hidden dimensions are sparser as measured by the percentage of nonzero entries.
Through extensive experiments we demonstrate that the emergence of sparsity is
a prevalent phenomenon that occurs for both natural language processing and
vision tasks, on both training and evaluation data, for Transformers of various
configurations, at layers of all depth levels, as well as for other
architectures including MLP-mixers and 2-layer MLPs. We show that sparsity also
emerges using training datasets with random labels, or with random inputs, or
with infinite amount of data, demonstrating that sparsity is not a result of a
specific family of datasets. We discuss how sparsity immediately implies a way
to significantly reduce the FLOP count and improve efficiency for Transformers.
Moreover, we demonstrate perhaps surprisingly that enforcing an even sparser
activation via Top-k thresholding with a small value of k brings a collection
of desired but missing properties for Transformers, namely less sensitivity to
noisy training data, more robustness to input corruptions, and better
calibration for their prediction confidence.
