  Transformers use the dense self-attention mechanism which gives a lot of
flexibility for long-range connectivity. Over multiple layers of a deep
transformer, the number of possible connectivity patterns increases
exponentially. However, very few of these contribute to the performance of the
network, and even fewer are essential. We hypothesize that there are sparsely
connected sub-networks within a transformer, called information pathways which
can be trained independently. However, the dynamic (i.e., input-dependent)
nature of these pathways makes it difficult to prune dense self-attention
during training. But the overall distribution of these pathways is often
predictable. We take advantage of this fact to propose Stochastically
Subsampled self-Attention (SSA) - a general-purpose training strategy for
transformers that can reduce both the memory and computational cost of
self-attention by 4 to 8 times during training while also serving as a
regularization method - improving generalization over dense training. We show
that an ensemble of sub-models can be formed from the subsampled pathways
within a network, which can achieve better performance than its densely
attended counterpart. We perform experiments on a variety of NLP, computer
vision and graph learning tasks in both generative and discriminative settings
to provide empirical evidence for our claims and show the effectiveness of the
proposed method.
