  We propose a method to automatically generate a domain- and task-adaptive
maskings of the given text for self-supervised pre-training, such that we can
effectively adapt the language model to a particular target task (e.g. question
answering). Specifically, we present a novel reinforcement learning-based
framework which learns the masking policy, such that using the generated masks
for further pre-training of the target language model helps improve task
performance on unseen texts. We use off-policy actor-critic with entropy
regularization and experience replay for reinforcement learning, and propose a
Transformer-based policy network that can consider the relative importance of
words in a given text. We validate our Neural Mask Generator (NMG) on several
question answering and text classification datasets using BERT and DistilBERT
as the language models, on which it outperforms rule-based masking strategies,
by automatically learning optimal adaptive maskings.
