  We propose an efficient inference procedure for non-autoregressive machine
translation that iteratively refines translation purely in the continuous
space. Given a continuous latent variable model for machine translation (Shu et
al., 2020), we train an inference network to approximate the gradient of the
marginal log probability of the target sentence, using only the latent variable
as input. This allows us to use gradient-based optimization to find the target
sentence at inference time that approximately maximizes its marginal
probability. As each refinement step only involves computation in the latent
space of low dimensionality (we use 8 in our experiments), we avoid
computational overhead incurred by existing non-autoregressive inference
procedures that often refine in token space. We compare our approach to a
recently proposed EM-like inference procedure (Shu et al., 2020) that optimizes
in a hybrid space, consisting of both discrete and continuous variables. We
evaluate our approach on WMT'14 En-De, WMT'16 Ro-En and IWSLT'16 De-En, and
observe two advantages over the EM-like inference: (1) it is computationally
efficient, i.e. each refinement step is twice as fast, and (2) it is more
effective, resulting in higher marginal probabilities and BLEU scores with the
same number of refinement steps. On WMT'14 En-De, for instance, our approach is
able to decode 6.2 times faster than the autoregressive model with minimal
degradation to translation quality (0.9 BLEU).
