  Word2vec is a popular family of algorithms for unsupervised training of dense
vector representations of words on large text corpuses. The resulting vectors
have been shown to capture semantic relationships among their corresponding
words, and have shown promise in reducing a number of natural language
processing (NLP) tasks to mathematical operations on these vectors. While
heretofore applications of word2vec have centered around vocabularies with a
few million words, wherein the vocabulary is the set of words for which vectors
are simultaneously trained, novel applications are emerging in areas outside of
NLP with vocabularies comprising several 100 million words. Existing word2vec
training systems are impractical for training such large vocabularies as they
either require that the vectors of all vocabulary words be stored in the memory
of a single server or suffer unacceptable training latency due to massive
network data transfer. In this paper, we present a novel distributed, parallel
training system that enables unprecedented practical training of vectors for
vocabularies with several 100 million words on a shared cluster of commodity
servers, using far less network traffic than the existing solutions. We
evaluate the proposed system on a benchmark dataset, showing that the quality
of vectors does not degrade relative to non-distributed training. Finally, for
several quarters, the system has been deployed for the purpose of matching
queries to ads in Gemini, the sponsored search advertising platform at Yahoo,
resulting in significant improvement of business metrics.
