  Vision transformer has demonstrated promising performance on challenging
computer vision tasks. However, directly training the vision transformers may
yield unstable and sub-optimal results. Recent works propose to improve the
performance of the vision transformers by modifying the transformer structures,
e.g., incorporating convolution layers. In contrast, we investigate an
orthogonal approach to stabilize the vision transformer training without
modifying the networks. We observe the instability of the training can be
attributed to the significant similarity across the extracted patch
representations. More specifically, for deep vision transformers, the
self-attention blocks tend to map different patches into similar latent
representations, yielding information loss and performance degradation. To
alleviate this problem, in this work, we introduce novel loss functions in
vision transformer training to explicitly encourage diversity across patch
representations for more discriminative feature extraction. We empirically show
that our proposed techniques stabilize the training and allow us to train wider
and deeper vision transformers. We further show the diversified features
significantly benefit the downstream tasks in transfer learning. For semantic
segmentation, we enhance the state-of-the-art (SOTA) results on Cityscapes and
ADE20k. Our code is available at
https://github.com/ChengyueGongR/PatchVisionTransformer.
