  Though achieving impressive results on many NLP tasks, the BERT-like masked
language models (MLM) encounter the discrepancy between pre-training and
inference. In light of this gap, we investigate the contextual representation
of pre-training and inference from the perspective of word probability
distribution. We discover that BERT risks neglecting the contextual word
similarity in pre-training. To tackle this issue, we propose an auxiliary gloss
regularizer module to BERT pre-training (GR-BERT), to enhance word semantic
similarity. By predicting masked words and aligning contextual embeddings to
corresponding glosses simultaneously, the word similarity can be explicitly
modeled. We design two architectures for GR-BERT and evaluate our model in
downstream tasks. Experimental results show that the gloss regularizer benefits
BERT in word-level and sentence-level semantic representation. The GR-BERT
achieves new state-of-the-art in lexical substitution task and greatly promotes
BERT sentence representation in both unsupervised and supervised STS tasks.
