  Post-training quantization~(PTQ) of transformer language models faces
significant challenges due to the existence of detrimental outliers in
activations. We observe that these outliers are concentrated in specific
channels and are asymmetric across channels. To address this issue, we propose
the Outlier Suppression+~(OS+) framework, which contains the channel-wise
shifting for asymmetry and channel-wise scaling for concentration. We show that
these operations can be seamlessly migrated into subsequent modules while
maintaining equivalence. Second, we propose a fast and stable scheme to
calculate effective shifting and scaling values. The channel-wise shifting
aligns the center of each channel for removal of outlier asymmetry. The
channel-wise scaling quantitatively evaluates changes brought by migration and
quantization for better quantization burden balance. We validate our OS+ under
both standard and fine-grained quantization settings with models including
BERT, OPT, BLOOM, BLOOMZ, and LLaMA. Comprehensive results across various tasks
demonstrate the superiority of our approach. Especially, with standard
quantization, OS+ can achieve near-floating-point performance on both small
models and large language models on 8-bit and 6-bit. Besides, we establish a
new state-of-the-art for 4-bit BERT with 15.5\% improvement. Our code is
available at \url{https://github.com/ModelTC/Outlier_Suppression_Plus}.
