  This paper describes the NiuTrans system for the WMT21 translation efficiency
task (http://statmt.org/wmt21/efficiency-task.html). Following last year's
work, we explore various techniques to improve efficiency while maintaining
translation quality. We investigate the combinations of lightweight Transformer
architectures and knowledge distillation strategies. Also, we improve the
translation efficiency with graph optimization, low precision, dynamic
batching, and parallel pre/post-processing. Our system can translate 247,000
words per second on an NVIDIA A100, being 3$\times$ faster than last year's
system. Our system is the fastest and has the lowest memory consumption on the
GPU-throughput track. The code, model, and pipeline will be available at
NiuTrans.NMT (https://github.com/NiuTrans/NiuTrans.NMT).
