  Federated learning frameworks typically require collaborators to share their
local gradient updates of a common model instead of sharing training data to
preserve privacy. However, prior works on Gradient Leakage Attacks showed that
private training data can be revealed from gradients. So far almost all
relevant works base their attacks on fully-connected or convolutional neural
networks. Given the recent overwhelmingly rising trend of adapting Transformers
to solve multifarious vision tasks, it is highly valuable to investigate the
privacy risk of vision transformers. In this paper, we analyse the gradient
leakage risk of self-attention based mechanism in both theoretical and
practical manners. Particularly, we propose APRIL - Attention PRIvacy Leakage,
which poses a strong threat to self-attention inspired models such as ViT.
Showing how vision Transformers are at the risk of privacy leakage via
gradients, we urge the significance of designing privacy-safer Transformer
models and defending schemes.
