  Zero-shot cross-lingual named entity recognition (NER) aims at transferring
knowledge from annotated and rich-resource data in source languages to
unlabeled and lean-resource data in target languages. Existing mainstream
methods based on the teacher-student distillation framework ignore the rich and
complementary information lying in the intermediate layers of pre-trained
language models, and domain-invariant information is easily lost during
transfer. In this study, a mixture of short-channel distillers (MSD) method is
proposed to fully interact the rich hierarchical information in the teacher
model and to transfer knowledge to the student model sufficiently and
efficiently. Concretely, a multi-channel distillation framework is designed for
sufficient information transfer by aggregating multiple distillers as a
mixture. Besides, an unsupervised method adopting parallel domain adaptation is
proposed to shorten the channels between the teacher and student models to
preserve domain-invariant features. Experiments on four datasets across nine
languages demonstrate that the proposed method achieves new state-of-the-art
performance on zero-shot cross-lingual NER and shows great generalization and
compatibility across languages and fields.
