  Transformer architecture has become the fundamental element of the widespread
natural language processing~(NLP) models. With the trends of large NLP models,
the increasing memory and computation costs hinder their efficient deployment
on resource-limited devices. Therefore, transformer quantization attracts wide
research interest. Recent work recognizes that structured outliers are the
critical bottleneck for quantization performance. However, their proposed
methods increase the computation overhead and still leave the outliers there.
To fundamentally address this problem, this paper delves into the inherent
inducement and importance of the outliers. We discover that $\boldsymbol
\gamma$ in LayerNorm (LN) acts as a sinful amplifier for the outliers, and the
importance of outliers varies greatly where some outliers provided by a few
tokens cover a large area but can be clipped sharply without negative impacts.
Motivated by these findings, we propose an outlier suppression framework
including two components: Gamma Migration and Token-Wise Clipping. The Gamma
Migration migrates the outlier amplifier to subsequent modules in an equivalent
transformation, contributing to a more quantization-friendly model without any
extra burden. The Token-Wise Clipping takes advantage of the large variance of
token range and designs a token-wise coarse-to-fine pipeline, obtaining a
clipping range with minimal final quantization loss in an efficient way. This
framework effectively suppresses the outliers and can be used in a
plug-and-play mode. Extensive experiments prove that our framework surpasses
the existing works and, for the first time, pushes the 6-bit post-training BERT
quantization to the full-precision (FP) level. Our code is available at
https://github.com/wimh966/outlier_suppression.
