  We present a Lipschitz continuous Transformer, called LipsFormer, to pursue
training stability both theoretically and empirically for Transformer-based
models. In contrast to previous practical tricks that address training
instability by learning rate warmup, layer normalization, attention
formulation, and weight initialization, we show that Lipschitz continuity is a
more essential property to ensure training stability. In LipsFormer, we replace
unstable Transformer component modules with Lipschitz continuous counterparts:
CenterNorm instead of LayerNorm, spectral initialization instead of Xavier
initialization, scaled cosine similarity attention instead of dot-product
attention, and weighted residual shortcut. We prove that these introduced
modules are Lipschitz continuous and derive an upper bound on the Lipschitz
constant of LipsFormer. Our experiments show that LipsFormer allows stable
training of deep Transformer architectures without the need of careful learning
rate tuning such as warmup, yielding a faster convergence and better
generalization. As a result, on the ImageNet 1K dataset, LipsFormer-Swin-Tiny
based on Swin Transformer training for 300 epochs can obtain 82.7\% without any
learning rate warmup. Moreover, LipsFormer-CSwin-Tiny, based on CSwin, training
for 300 epochs achieves a top-1 accuracy of 83.5\% with 4.7G FLOPs and 24M
parameters. The code will be released at
\url{https://github.com/IDEA-Research/LipsFormer}.
