  The transformer is the most critical algorithm innovation of the Nature
Language Processing (NLP) field in recent years. Unlike the Recurrent Neural
Network (RNN) models, Transformers can process on dimensions of sequence
lengths in parallel, therefore leading to better accuracy on long sequences.
However, efficient deployments of them for online services in data centers
equipped with GPUs are not easy. First, more computation introduced by
transformer structures makes it more challenging to meet the latency and
throughput constraints of serving. Second, NLP tasks take in sentences of
variable length. The variability of input dimensions brings a severe problem to
efficient memory management and serving optimization.
  This paper designed a transformer serving system called TurboTransformers,
which consists of a computing runtime and a serving framework to solve the
above challenges. Three innovative features make it stand out from other
similar works. An efficient parallel algorithm is proposed for GPU-based batch
reduction operations, like Softmax and LayerNorm, major hot spots besides BLAS
routines. A memory allocation algorithm, which better balances the memory
footprint and allocation/free efficiency, is designed for variable-length input
situations. A serving framework equipped with a new batch scheduler using
dynamic programming achieves the optimal throughput on variable-length
requests. The system can achieve the state-of-the-art transformer model serving
performance on GPU platforms and can be seamlessly integrated into your PyTorch
code with a few lines of code.
