  It is difficult for non-autoregressive translation (NAT) models to capture
the multi-modal distribution of target translations due to their conditional
independence assumption, which is known as the "multi-modality problem",
including the lexical multi-modality and the syntactic multi-modality. While
the first one has been well studied, the syntactic multi-modality brings severe
challenge to the standard cross entropy (XE) loss in NAT and is under studied.
In this paper, we conduct a systematic study on the syntactic multi-modality
problem. Specifically, we decompose it into short- and long-range syntactic
multi-modalities and evaluate several recent NAT algorithms with advanced loss
functions on both carefully designed synthesized datasets and real datasets. We
find that the Connectionist Temporal Classification (CTC) loss and the
Order-Agnostic Cross Entropy (OAXE) loss can better handle short- and
long-range syntactic multi-modalities respectively. Furthermore, we take the
best of both and design a new loss function to better handle the complicated
syntactic multi-modality in real-world datasets. To facilitate practical usage,
we provide a guide to use different loss functions for different kinds of
syntactic multi-modality.
