  Most popular word embedding techniques involve implicit or explicit
factorization of a word co-occurrence based matrix into low rank factors. In
this paper, we aim to generalize this trend by using numerical methods to
factor higher-order word co-occurrence based arrays, or \textit{tensors}. We
present four word embeddings using tensor factorization and analyze their
advantages and disadvantages. One of our main contributions is a novel joint
symmetric tensor factorization technique related to the idea of coupled tensor
factorization. We show that embeddings based on tensor factorization can be
used to discern the various meanings of polysemous words without being
explicitly trained to do so, and motivate the intuition behind why this works
in a way that doesn't with existing methods. We also modify an existing word
embedding evaluation metric known as Outlier Detection [Camacho-Collados and
Navigli, 2016] to evaluate the quality of the order-$N$ relations that a word
embedding captures, and show that tensor-based methods outperform existing
matrix-based methods at this task. Experimentally, we show that all of our word
embeddings either outperform or are competitive with state-of-the-art baselines
commonly used today on a variety of recent datasets. Suggested applications of
tensor factorization-based word embeddings are given, and all source code and
pre-trained vectors are publicly available online.
