  The negative sampling (NEG) objective function, used in word2vec, is a
simplification of the Noise Contrastive Estimation (NCE) method. NEG was found
to be highly effective in learning continuous word representations. However,
unlike NCE, it was considered inapplicable for the purpose of learning the
parameters of a language model. In this study, we refute this assertion by
providing a principled derivation for NEG-based language modeling, founded on a
novel analysis of a low-dimensional approximation of the matrix of pointwise
mutual information between the contexts and the predicted words. The obtained
language modeling is closely related to NCE language models but is based on a
simplified objective function. We thus provide a unified formulation for two
main language processing tasks, namely word embedding and language modeling,
based on the NEG objective function. Experimental results on two popular
language modeling benchmarks show comparable perplexity results, with a small
advantage to NEG over NCE.
