  This paper takes a step towards theoretical analysis of the relationship
between word embeddings and context embeddings in models such as word2vec. We
start from basic probabilistic assumptions on the nature of word vectors,
context vectors, and text generation. These assumptions are well supported
either empirically or theoretically by the existing literature. Next, we show
that under these assumptions the widely-used word-word PMI matrix is
approximately a random symmetric Gaussian ensemble. This, in turn, implies that
context vectors are reflections of word vectors in approximately half the
dimensions. As a direct application of our result, we suggest a theoretically
grounded way of tying weights in the SGNS model.
