  In this paper, we propose Shallow Aggressive Decoding (SAD) to improve the
online inference efficiency of the Transformer for instantaneous Grammatical
Error Correction (GEC). SAD optimizes the online inference efficiency for GEC
by two innovations: 1) it aggressively decodes as many tokens as possible in
parallel instead of always decoding only one token in each step to improve
computational parallelism; 2) it uses a shallow decoder instead of the
conventional Transformer architecture with balanced encoder-decoder depth to
reduce the computational cost during inference. Experiments in both English and
Chinese GEC benchmarks show that aggressive decoding could yield the same
predictions as greedy decoding but with a significant speedup for online
inference. Its combination with the shallow decoder could offer an even higher
online inference speedup over the powerful Transformer baseline without quality
loss. Not only does our approach allow a single model to achieve the
state-of-the-art results in English GEC benchmarks: 66.4 F0.5 in the CoNLL-14
and 72.9 F0.5 in the BEA-19 test set with an almost 10x online inference
speedup over the Transformer-big model, but also it is easily adapted to other
languages. Our code is available at
https://github.com/AutoTemp/Shallow-Aggressive-Decoding.
