  Recently many plug-and-play self-attention modules (SAMs) are proposed to
enhance the model generalization by exploiting the internal information of deep
convolutional neural networks (CNNs). In general, previous works ignore where
to plug in the SAMs since they connect the SAMs individually with each block of
the entire CNN backbone for granted, leading to incremental computational cost
and the number of parameters with the growth of network depth. However, we
empirically find and verify some counterintuitive phenomena that: (a)
Connecting the SAMs to all the blocks may not always bring the largest
performance boost, and connecting to partial blocks would be even better; (b)
Adding the SAMs to a CNN may not always bring a performance boost, and instead
it may even harm the performance of the original CNN backbone. Therefore, we
articulate and demonstrate the Lottery Ticket Hypothesis for Self-attention
Networks: a full self-attention network contains a subnetwork with sparse
self-attention connections that can (1) accelerate inference, (2) reduce extra
parameter increment, and (3) maintain accuracy. In addition to the empirical
evidence, this hypothesis is also supported by our theoretical evidence.
Furthermore, we propose a simple yet effective reinforcement-learning-based
method to search the ticket, i.e., the connection scheme that satisfies the
three above-mentioned conditions. Extensive experiments on widely-used
benchmark datasets and popular self-attention networks show the effectiveness
of our method. Besides, our experiments illustrate that our searched ticket has
the capacity of transferring to some vision tasks, e.g., crowd counting and
segmentation.
