  Conditional masked language model (CMLM) training has proven successful for
non-autoregressive and semi-autoregressive sequence generation tasks, such as
machine translation. Given a trained CMLM, however, it is not clear what the
best inference strategy is. We formulate masked inference as a factorization of
conditional probabilities of partial sequences, show that this does not harm
performance, and investigate a number of simple heuristics motivated by this
perspective. We identify a thresholding strategy that has advantages over the
standard "mask-predict" algorithm, and provide analyses of its behavior on
machine translation tasks.
