  Pre-trained neural language models bring significant improvement for various
NLP tasks, by fine-tuning the models on task-specific training sets. During
fine-tuning, the parameters are initialized from pre-trained models directly,
which ignores how the learning process of similar NLP tasks in different
domains is correlated and mutually reinforced. In this paper, we propose an
effective learning procedure named Meta Fine-Tuning (MFT), served as a
meta-learner to solve a group of similar NLP tasks for neural language models.
Instead of simply multi-task training over all the datasets, MFT only learns
from typical instances of various domains to acquire highly transferable
knowledge. It further encourages the language model to encode domain-invariant
representations by optimizing a series of novel domain corruption loss
functions. After MFT, the model can be fine-tuned for each domain with better
parameter initializations and higher generalization ability. We implement MFT
upon BERT to solve several multi-domain text mining tasks. Experimental results
confirm the effectiveness of MFT and its usefulness for few-shot learning.
