  The latest industrial inference engines, such as FasterTransformer and
TurboTransformers, have verified that half-precision floating point (FP16) and
8-bit integer (INT8) quantization can greatly improve model inference speed.
However, the existing INT8 quantization methods are too complicated, and
improper usage will lead to model performance damage greatly. In this paper, we
develop a toolkit for users to easily quantize their models for inference, in
which Self-Adaptive Mixed-Precision (SAMP) is proposed to automatically control
quantization rate by a mixed-precision architecture to balance model accuracy
and efficiency. Experimental results show that our SAMP toolkit has a higher
speedup than PyTorch and FasterTransformer while ensuring the required
accuracy. In addition, SAMP is based on a modular design, decoupling the
tokenizer, embedding, encoder and target layers, which allows users to handle
various downstream tasks and can be seamlessly integrated into PyTorch.
