  Albeit the universal representational power of pre-trained language models,
adapting them onto a specific NLP task still requires a considerably large
amount of labeled data. Effective task fine-tuning meets challenges when only a
few labeled examples are present for the task. In this paper, we aim to the
address of the problem of few shot task learning by exploiting and transferring
from a different task which admits a related but disparate label space.
Specifically, we devise a label transfer network (LTN) to transform the labels
from source task to the target task of interest for training. Both the LTN and
the model for task prediction are learned via a bi-level optimization
framework, which we term as MetaXT. MetaXT offers a principled solution to best
adapt a pre-trained language model to the target task by transferring knowledge
from the source task. Empirical evaluations on cross-task transfer settings for
four NLP tasks, from two different types of label space disparities,
demonstrate the effectiveness of MetaXT, especially when the labeled data in
the target task is limited.
