  Current NLP models are predominantly trained through a two-stage "pre-train
then fine-tune" pipeline. Prior work has shown that inserting an intermediate
pre-training stage, using heuristic masking policies for masked language
modeling (MLM), can significantly improve final performance. However, it is
still unclear (1) in what cases such intermediate pre-training is helpful, (2)
whether hand-crafted heuristic objectives are optimal for a given task, and (3)
whether a masking policy designed for one task is generalizable beyond that
task. In this paper, we perform a large-scale empirical study to investigate
the effect of various masking policies in intermediate pre-training with nine
selected tasks across three categories. Crucially, we introduce methods to
automate the discovery of optimal masking policies via direct supervision or
meta-learning. We conclude that the success of intermediate pre-training is
dependent on appropriate pre-train corpus, selection of output format (i.e.,
masked spans or full sentence), and clear understanding of the role that MLM
plays for the downstream task. In addition, we find our learned masking
policies outperform the heuristic of masking named entities on TriviaQA, and
policies learned from one task can positively transfer to other tasks in
certain cases, inviting future research in this direction.
