  Uncontextualized word embeddings are reliable feature representations of
words used to obtain high quality results for various NLP applications. Given
the historical success of word embeddings in NLP, we propose a retrospective on
some of the most well-known word embedding algorithms. In this work, we
deconstruct Word2vec, GloVe, and others, into a common form, unveiling some of
the necessary and sufficient conditions required for making performant word
embeddings. We find that each algorithm: (1) fits vector-covector dot products
to approximate pointwise mutual information (PMI); and, (2) modulates the loss
gradient to balance weak and strong signals. We demonstrate that these two
algorithmic features are sufficient conditions to construct a novel word
embedding algorithm, Hilbert-MLE. We find that its embeddings obtain equivalent
or better performance against other algorithms across 17 intrinsic and
extrinsic datasets.
