  The Transformer has emerged as a versatile and effective architecture with
broad applications. However, it still remains an open problem how to
efficiently train a Transformer model of high utility with differential privacy
guarantees. In this paper, we identify two key challenges in learning
differentially private Transformers, i.e., heavy computation overhead due to
per-sample gradient clipping and unintentional attention distraction within the
attention mechanism. In response, we propose DPFormer, equipped with Phantom
Clipping and Re-Attention Mechanism, to address these challenges. Our
theoretical analysis shows that DPFormer can reduce computational costs during
gradient clipping and effectively mitigate attention distraction (which could
obstruct the training process and lead to a significant performance drop,
especially in the presence of long-tailed data). Such analysis is further
corroborated by empirical results on two real-world datasets, demonstrating the
efficiency and effectiveness of the proposed DPFormer.
