  We present multilingual Pre-trained Machine Reader (mPMR), a novel method for
multilingual machine reading comprehension (MRC)-style pre-training. mPMR aims
to guide multilingual pre-trained language models (mPLMs) to perform natural
language understanding (NLU) including both sequence classification and span
extraction in multiple languages. To achieve cross-lingual generalization when
only source-language fine-tuning data is available, existing mPLMs solely
transfer NLU capability from a source language to target languages. In
contrast, mPMR allows the direct inheritance of multilingual NLU capability
from the MRC-style pre-training to downstream tasks. Therefore, mPMR acquires
better NLU capability for target languages. mPMR also provides a unified solver
for tackling cross-lingual span extraction and sequence classification, thereby
enabling the extraction of rationales to explain the sentence-pair
classification process.
