  Masked language modeling is widely used for pretraining large language models
for natural language understanding (NLU). However, random masking is
suboptimal, allocating an equal masking rate for all tokens. In this paper, we
propose InforMask, a new unsupervised masking strategy for training masked
language models. InforMask exploits Pointwise Mutual Information (PMI) to
select the most informative tokens to mask. We further propose two
optimizations for InforMask to improve its efficiency. With a one-off
preprocessing step, InforMask outperforms random masking and previously
proposed masking strategies on the factual recall benchmark LAMA and the
question answering benchmark SQuAD v1 and v2.
