  Distributed representations of words encode lexical semantic information, but
what type of information is encoded and how? Focusing on the skip-gram with
negative-sampling method, we found that the squared norm of static word
embedding encodes the information gain conveyed by the word; the information
gain is defined by the Kullback-Leibler divergence of the co-occurrence
distribution of the word to the unigram distribution. Our findings are
explained by the theoretical framework of the exponential family of probability
distributions and confirmed through precise experiments that remove spurious
correlations arising from word frequency. This theory also extends to
contextualized word embeddings in language models or any neural networks with
the softmax output layer. We also demonstrate that both the KL divergence and
the squared norm of embedding provide a useful metric of the informativeness of
a word in tasks such as keyword extraction, proper-noun discrimination, and
hypernym discrimination.
