  Non-autoregressive Transformer (NAT) is a family of text generation models,
which aims to reduce the decoding latency by predicting the whole sentences in
parallel. However, such latency reduction sacrifices the ability to capture
left-to-right dependencies, thereby making NAT learning very challenging. In
this paper, we present theoretical and empirical analyses to reveal the
challenges of NAT learning and propose a unified perspective to understand
existing successes. First, we show that simply training NAT by maximizing the
likelihood can lead to an approximation of marginal distributions but drops all
dependencies between tokens, where the dropped information can be measured by
the dataset's conditional total correlation. Second, we formalize many previous
objectives in a unified framework and show that their success can be concluded
as maximizing the likelihood on a proxy distribution, leading to a reduced
information loss. Empirical studies show that our perspective can explain the
phenomena in NAT learning and guide the design of new training methods.
