  Transformer-based models, such as BERT, have been widely applied in a wide
range of natural language processing tasks. However, one inevitable side effect
is that they require massive memory storage and inference cost when deployed in
production. Quantization is one of the popularized ways to alleviate the cost.
However, the previous 8-bit quantization strategy based on INT8 data format
either suffers from the degradation of accuracy in a Post-Training Quantization
(PTQ) fashion or requires an expensive Quantization-Aware Training (QAT)
process. Recently, a new numeric format FP8 (i.e. floating-point of 8-bits) has
been proposed and supported in commercial AI computing platforms such as H100.
In this paper, we empirically validate the effectiveness of FP8 as a way to do
Post-Training Quantization without significant loss of accuracy, with a simple
calibration and format conversion process. We adopt the FP8 standard proposed
by NVIDIA Corp. (2022) in our extensive experiments of BERT variants on GLUE
and SQuAD v1.1 datasets, and show that PTQ with FP8 can significantly improve
the accuracy upon that with INT8, to the extent of the full-precision model.
