  Transformer-based autoregressive (AR) methods have achieved appealing
performance for varied sequence-to-sequence generation tasks, e.g., neural
machine translation, summarization, and code generation, but suffer from low
inference efficiency. To speed up the inference stage, many non-autoregressive
(NAR) strategies have been proposed in the past few years. Among them, the
conditional masked language model (CMLM) is one of the most versatile
frameworks, as it can support many different sequence generation scenarios and
achieve very competitive performance on these tasks. In this paper, we further
introduce a simple yet effective adaptive masking over masking strategy to
enhance the refinement capability of the decoder and make the encoder
optimization easier. Experiments on \textbf{3} different tasks (neural machine
translation, summarization, and code generation) with \textbf{15} datasets in
total confirm that our proposed simple method achieves significant performance
improvement over the strong CMLM model. Surprisingly, our proposed model yields
state-of-the-art performance on neural machine translation (\textbf{34.62} BLEU
on WMT16 EN$\to$RO, \textbf{34.82} BLEU on WMT16 RO$\to$EN, and \textbf{34.84}
BLEU on IWSLT De$\to$En) and even better performance than the \textbf{AR}
Transformer on \textbf{7} benchmark datasets with at least \textbf{2.2$\times$}
speedup. Our code is available at GitHub.
