  Due to the highly parallelizable architecture, Transformer is faster to train
than RNN-based models and popularly used in machine translation tasks. However,
at inference time, each output word requires all the hidden states of the
previously generated words, which limits the parallelization capability, and
makes it much slower than RNN-based ones. In this paper, we systematically
analyze the time cost of different components of both the Transformer and
RNN-based model. Based on it, we propose a hybrid network of self-attention and
RNN structures, in which, the highly parallelizable self-attention is utilized
as the encoder, and the simpler RNN structure is used as the decoder. Our
hybrid network can decode 4-times faster than the Transformer. In addition,
with the help of knowledge distillation, our hybrid network achieves comparable
translation quality to the original Transformer.
