  Neural text generation models are typically trained by maximizing
log-likelihood with the sequence cross entropy (CE) loss, which encourages an
exact token-by-token match between a target sequence with a generated sequence.
Such training objective is sub-optimal when the target sequence is not perfect,
e.g., when the target sequence is corrupted with noises, or when only weak
sequence supervision is available. To address the challenge, we propose a novel
Edit-Invariant Sequence Loss (EISL), which computes the matching loss of a
target n-gram with all n-grams in the generated sequence. EISL is designed to
be robust to various noises and edits in the target sequences. Moreover, the
EISL computation is essentially an approximate convolution operation with
target n-grams as kernels, which is easy to implement and efficient to compute
with existing libraries. To demonstrate the effectiveness of EISL, we conduct
experiments on a wide range of tasks, including machine translation with noisy
target sequences, unsupervised text style transfer with only weak training
signals, and non-autoregressive generation with non-predefined generation
order. Experimental results show our method significantly outperforms the
common CE loss and other strong baselines on all the tasks. EISL has a simple
API that can be used as a drop-in replacement of the CE loss:
https://github.com/guangyliu/EISL.
