  We experiment with new methods for learning how related words are positioned
relative to each other in word embedding spaces. Previous approaches learned
constant vector offsets: vectors that point from source tokens to target tokens
with an assumption that these offsets were parallel to each other. We show that
the offsets between related tokens are closer to orthogonal than parallel, and
that they have low cosine similarities. We proceed by making a different
assumption; target tokens are linearly separable from source and un-labeled
tokens. We show that a max-margin hyperplane can separate target tokens and
that vectors orthogonal to this hyperplane represent the relationship between
source and targets. We find that this representation of the relationship
obtains the best results in dis-covering linguistic regularities. We experiment
with vector space models trained by a variety of algorithms (Word2vec:
CBOW/skip-gram, fastText, or GloVe), and various word context choices such as
linear word-order, syntax dependency grammars, and with and without knowledge
of word position. These experiments show that our model, SVMCos, is robust to a
range of experimental choices when training word embeddings.
