  Word2Vec is the most popular model for word representation and has been
widely investigated in literature. However, its noise distribution for negative
sampling is decided by empirical trials and the optimality has always been
ignored. We suggest that the distribution is a sub-optimal choice, and propose
to use a sub-sampled unigram distribution for better negative sampling. Our
contributions include: (1) proposing the concept of semantics quantification
and deriving a suitable sub-sampling rate for the proposed distribution
adaptive to different training corpora; (2) demonstrating the advantages of our
approach in both negative sampling and noise contrastive estimation by
extensive evaluation tasks; and (3) proposing a semantics weighted model for
the MSR sentence completion task, resulting in considerable improvements. Our
work not only improves the quality of word vectors but also benefits current
understanding of Word2Vec.
