  Word alignment, which aims to align translationally equivalent words between
source and target sentences, plays an important role in many natural language
processing tasks. Current unsupervised neural alignment methods focus on
inducing alignments from neural machine translation models, which does not
leverage the full context in the target sequence. In this paper, we propose
Mask-Align, a self-supervised word alignment model that takes advantage of the
full context on the target side. Our model masks out each target token and
predicts it conditioned on both source and the remaining target tokens. This
two-step process is based on the assumption that the source token contributing
most to recovering the masked target token should be aligned. We also introduce
an attention variant called leaky attention, which alleviates the problem of
unexpected high cross-attention weights on special tokens such as periods.
Experiments on four language pairs show that our model outperforms previous
unsupervised neural aligners and obtains new state-of-the-art results.
