  With the development of deep learning and Transformer-based pre-trained
models like BERT, the accuracy of many NLP tasks has been dramatically
improved. However, the large number of parameters and computations also pose
challenges for their deployment. For instance, using BERT can improve the
predictions in the financial sentiment analysis (FSA) task but slow it down,
where speed and accuracy are equally important in terms of profits. To address
these issues, we first propose an efficient and lightweight BERT (ELBERT) along
with a novel confidence-window-based (CWB) early exit mechanism. Based on
ELBERT, an innovative method to accelerate text processing on the GPU platform
is developed, solving the difficult problem of making the early exit mechanism
work more effectively with a large input batch size. Afterward, a fast and
high-accuracy FSA system is built. Experimental results show that the proposed
CWB early exit mechanism achieves significantly higher accuracy than existing
early exit methods on BERT under the same computation cost. By using this
acceleration method, our FSA system can boost the processing speed by nearly 40
times to over 1000 texts per second with sufficient accuracy, which is nearly
twice as fast as FastBERT, thus providing a more powerful text processing
capability for modern trading systems.
