  Word embeddings generated by neural network methods such as word2vec (W2V)
are well known to exhibit seemingly linear behaviour, e.g. the embeddings of
analogy "woman is to queen as man is to king" approximately describe a
parallelogram. This property is particularly intriguing since the embeddings
are not trained to achieve it. Several explanations have been proposed, but
each introduces assumptions that do not hold in practice. We derive a
probabilistically grounded definition of paraphrasing that we re-interpret as
word transformation, a mathematical description of "$w_x$ is to $w_y$". From
these concepts we prove existence of linear relationships between W2V-type
embeddings that underlie the analogical phenomenon, identifying explicit error
terms.
