  GloVe learns word embeddings by leveraging statistical information from word
co-occurrence matrices. However, word pairs in the matrices are extracted from
a predefined local context window, which might lead to limited word pairs and
potentially semantic irrelevant word pairs. In this paper, we propose SemGloVe,
which distills semantic co-occurrences from BERT into static GloVe word
embeddings. Particularly, we propose two models to extract co-occurrence
statistics based on either the masked language model or the multi-head
attention weights of BERT. Our methods can extract word pairs without limiting
by the local window assumption and can define the co-occurrence weights by
directly considering the semantic distance between word pairs. Experiments on
several word similarity datasets and four external tasks show that SemGloVe can
outperform GloVe.
