  Learning multilingual representations of text has proven a successful method
for many cross-lingual transfer learning tasks. There are two main paradigms
for learning such representations: (1) alignment, which maps different
independently trained monolingual representations into a shared space, and (2)
joint training, which directly learns unified multilingual representations
using monolingual and cross-lingual objectives jointly. In this paper, we first
conduct direct comparisons of representations learned using both of these
methods across diverse cross-lingual tasks. Our empirical results reveal a set
of pros and cons for both methods, and show that the relative performance of
alignment versus joint training is task-dependent. Stemming from this analysis,
we propose a simple and novel framework that combines these two previously
mutually-exclusive approaches. Extensive experiments demonstrate that our
proposed framework alleviates limitations of both approaches, and outperforms
existing methods on the MUSE bilingual lexicon induction (BLI) benchmark. We
further show that this framework can generalize to contextualized
representations such as Multilingual BERT, and produces state-of-the-art
results on the CoNLL cross-lingual NER benchmark.
