  Multilingual BERT (mBERT), a language model pre-trained on large multilingual
corpora, has impressive zero-shot cross-lingual transfer capabilities and
performs surprisingly well on zero-shot POS tagging and Named Entity
Recognition (NER), as well as on cross-lingual model transfer. At present, the
mainstream methods to solve the cross-lingual downstream tasks are always using
the last transformer layer's output of mBERT as the representation of
linguistic information. In this work, we explore the complementary property of
lower layers to the last transformer layer of mBERT. A feature aggregation
module based on an attention mechanism is proposed to fuse the information
contained in different layers of mBERT. The experiments are conducted on four
zero-shot cross-lingual transfer datasets, and the proposed method obtains
performance improvements on key multilingual benchmark tasks XNLI (+1.5 %),
PAWS-X (+2.4 %), NER (+1.2 F1), and POS (+1.5 F1). Through the analysis of the
experimental results, we prove that the layers before the last layer of mBERT
can provide extra useful information for cross-lingual downstream tasks and
explore the interpretability of mBERT empirically.
