  Static and contextual multilingual embeddings have complementary strengths.
Static embeddings, while less expressive than contextual language models, can
be more straightforwardly aligned across multiple languages. We combine the
strengths of static and contextual models to improve multilingual
representations. We extract static embeddings for 40 languages from XLM-R,
validate those embeddings with cross-lingual word retrieval, and then align
them using VecMap. This results in high-quality, highly multilingual static
embeddings. Then we apply a novel continued pre-training approach to XLM-R,
leveraging the high quality alignment of our static embeddings to better align
the representation space of XLM-R. We show positive results for multiple
complex semantic tasks. We release the static embeddings and the continued
pre-training code. Unlike most previous work, our continued pre-training
approach does not require parallel text.
