  Non-autoregressive models generate target words in a parallel way, which
achieve a faster decoding speed but at the sacrifice of translation accuracy.
To remedy a flawed translation by non-autoregressive models, a promising
approach is to train a conditional masked translation model (CMTM), and refine
the generated results within several iterations. Unfortunately, such approach
hardly considers the \textit{sequential dependency} among target words, which
inevitably results in a translation degradation. Hence, instead of solely
training a Transformer-based CMTM, we propose a Self-Review Mechanism to infuse
sequential information into it. Concretely, we insert a left-to-right mask to
the same decoder of CMTM, and then induce it to autoregressively review whether
each generated word from CMTM is supposed to be replaced or kept. The
experimental results (WMT14 En$\leftrightarrow$De and WMT16
En$\leftrightarrow$Ro) demonstrate that our model uses dramatically less
training computations than the typical CMTM, as well as outperforms several
state-of-the-art non-autoregressive models by over 1 BLEU. Through knowledge
distillation, our model even surpasses a typical left-to-right Transformer
model, while significantly speeding up decoding.
