  Adapter-tuning is a paradigm that transfers a pretrained language model to
downstream tasks by adding and tuning a small number of new parameters.
Previously proposed adapter architectures are all feed-forward neural networks.
In this paper, we investigate the effectiveness of using tiny-attention --
i.e., attention with extremely small per-head dimensionality -- as adapters.
Our tiny-attention adapter learns to modify the hidden states at each position
directly conditioned on the hidden states at all the other positions, which is
missed by the previously proposed adapters. Moreover, we view its multiple
attention heads as a mixture of experts and propose to average their weights
during deployment, which further reduces its inference computation cost. On the
GLUE benchmark, our tiny-attention adapter outperforms the other
parameter-efficient transfer learning methods as well as full fine-tuning while
only updating 0.05% of the parameters. On the FewGLUE benchmark, its
performance is comparable to that of GPT-3 and PET.
