  Multilingual BERT (mBERT), XLM-RoBERTa (XLMR) and other unsupervised
multilingual encoders can effectively learn cross-lingual representation.
Explicit alignment objectives based on bitexts like Europarl or MultiUN have
been shown to further improve these representations. However, word-level
alignments are often suboptimal and such bitexts are unavailable for many
languages. In this paper, we propose a new contrastive alignment objective that
can better utilize such signal, and examine whether these previous alignment
methods can be adapted to noisier sources of aligned data: a randomly sampled 1
million pair subset of the OPUS collection. Additionally, rather than report
results on a single dataset with a single model run, we report the mean and
standard derivation of multiple runs with different seeds, on four datasets and
tasks. Our more extensive analysis finds that, while our new objective
outperforms previous work, overall these methods do not improve performance
with a more robust evaluation framework. Furthermore, the gains from using a
better underlying model eclipse any benefits from alignment training. These
negative results dictate more care in evaluating these methods and suggest
limitations in applying explicit alignment objectives.
