  Non-AutoRegressive (NAR) text generation models have drawn much attention
because of their significantly faster decoding speed and good generation
quality in machine translation. However, in a wider range of text generation
tasks, existing NAR models lack proper pre-training, making them still far
behind the pre-trained autoregressive models. In this paper, we propose
Pre-trained Directed Acyclic Transformer (PreDAT) and a novel pre-training task
to promote prediction consistency in NAR generation. Experiments on five text
generation tasks show that our PreDAT remarkably outperforms existing
pre-trained NAR models (+4.2 scores on average) and even achieves better
results than pre-trained autoregressive baselines in n-gram-based metrics,
along with 17 times speedup in throughput. Further analysis shows that PreDAT
benefits from the unbiased prediction order that alleviates the error
accumulation problem in autoregressive generation, which provides new insights
into the advantages of NAR generation.
