  Cross-lingual transfer (CLT) is of various applications. However, labeled
cross-lingual corpus is expensive or even inaccessible, especially in the
fields where labels are private, such as diagnostic results of symptoms in
medicine and user profiles in business. Nevertheless, there are off-the-shelf
models in these sensitive fields. Instead of pursuing the original labels, a
workaround for CLT is to transfer knowledge from the off-the-shelf models
without labels. To this end, we define a novel CLT problem named FreeTransfer-X
that aims to achieve knowledge transfer from the off-the-shelf models in
rich-resource languages. To address the problem, we propose a 2-step knowledge
distillation (KD, Hinton et al., 2015) framework based on multilingual
pre-trained language models (mPLM). The significant improvement over strong
neural machine translation (NMT) baselines demonstrates the effectiveness of
the proposed method. In addition to reducing annotation cost and protecting
private labels, the proposed method is compatible with different networks and
easy to be deployed. Finally, a range of analyses indicate the great potential
of the proposed method.
