  Transformer based models, like BERT and RoBERTa, have achieved
state-of-the-art results in many Natural Language Processing tasks. However,
their memory footprint, inference latency, and power consumption are
prohibitive efficient inference at the edge, and even at the data center. While
quantization can be a viable solution for this, previous work on quantizing
Transformer based models use floating-point arithmetic during inference, which
cannot efficiently utilize integer-only logical units such as the recent Turing
Tensor Cores, or traditional integer-only ARM processors. In this work, we
propose I-BERT, a novel quantization scheme for Transformer based models that
quantizes the entire inference with integer-only arithmetic. Based on
lightweight integer-only approximation methods for nonlinear operations, e.g.,
GELU, Softmax, and Layer Normalization, I-BERT performs an end-to-end
integer-only BERT inference without any floating point calculation. We evaluate
our approach on GLUE downstream tasks using RoBERTa-Base/Large. We show that
for both cases, I-BERT achieves similar (and slightly higher) accuracy as
compared to the full-precision baseline. Furthermore, our preliminary
implementation of I-BERT shows a speedup of 2.4-4.0x for INT8 inference on a T4
GPU system as compared to FP32 inference. The framework has been developed in
PyTorch and has been open-sourced.
