  Recently, pre-trained language models have achieved remarkable success in a
broad range of natural language processing tasks. However, in multilingual
setting, it is extremely resource-consuming to pre-train a deep language model
over large-scale corpora for each language. Instead of exhaustively
pre-training monolingual language models independently, an alternative solution
is to pre-train a powerful multilingual deep language model over large-scale
corpora in hundreds of languages. However, the vocabulary size for each
language in such a model is relatively small, especially for low-resource
languages. This limitation inevitably hinders the performance of these
multilingual models on tasks such as sequence labeling, wherein in-depth
token-level or sentence-level understanding is essential.
  In this paper, inspired by previous methods designed for monolingual
settings, we investigate two approaches (i.e., joint mapping and mixture
mapping) based on a pre-trained multilingual model BERT for addressing the
out-of-vocabulary (OOV) problem on a variety of tasks, including part-of-speech
tagging, named entity recognition, machine translation quality estimation, and
machine reading comprehension. Experimental results show that using mixture
mapping is more promising. To the best of our knowledge, this is the first work
that attempts to address and discuss the OOV issue in multilingual settings.
