  Knowledge distillation (KD) is commonly used to construct synthetic data for
training non-autoregressive translation (NAT) models. However, there exists a
discrepancy on low-frequency words between the distilled and the original data,
leading to more errors on predicting low-frequency words. To alleviate the
problem, we directly expose the raw data into NAT by leveraging pretraining. By
analyzing directed alignments, we found that KD makes low-frequency source
words aligned with targets more deterministically but fails to align sufficient
low-frequency words from target to source. Accordingly, we propose reverse KD
to rejuvenate more alignments for low-frequency target words. To make the most
of authentic and synthetic data, we combine these complementary approaches as a
new training strategy for further boosting NAT performance. We conduct
experiments on five translation benchmarks over two advanced architectures.
Results demonstrate that the proposed approach can significantly and
universally improve translation quality by reducing translation errors on
low-frequency words. Encouragingly, our approach achieves 28.2 and 33.9 BLEU
points on the WMT14 English-German and WMT16 Romanian-English datasets,
respectively. Our code, data, and trained models are available at
\url{https://github.com/alphadl/RLFW-NAT}.
