  We analyze the masked language modeling pretraining objective function from
the perspective of the distributional hypothesis. We investigate whether better
sample efficiency and the better generalization capability of models pretrained
with masked language modeling can be attributed to the semantic similarity
encoded in the pretraining data's distributional property. Via a synthetic
dataset, our analysis suggests that distributional property indeed leads to the
better sample efficiency of pretrained masked language models, but does not
fully explain the generalization capability. We also conduct analyses over two
real-world datasets and demonstrate that the distributional property does not
explain the generalization ability of pretrained natural language models
either. Our results illustrate our limited understanding of model pretraining
and provide future research directions.
