  Non-autoregressive Transformers (NATs) significantly reduce the decoding
latency by generating all tokens in parallel. However, such independent
predictions prevent NATs from capturing the dependencies between the tokens for
generating multiple possible translations. In this paper, we propose Directed
Acyclic Transfomer (DA-Transformer), which represents the hidden states in a
Directed Acyclic Graph (DAG), where each path of the DAG corresponds to a
specific translation. The whole DAG simultaneously captures multiple
translations and facilitates fast predictions in a non-autoregressive fashion.
Experiments on the raw training data of WMT benchmark show that DA-Transformer
substantially outperforms previous NATs by about 3 BLEU on average, which is
the first NAT model that achieves competitive results with autoregressive
Transformers without relying on knowledge distillation.
