  Co-occurrences between two words provide useful insights into the semantics
of those words. Consequently, numerous prior work on word embedding learning
have used co-occurrences between two words as the training signal for learning
word embeddings. However, in natural language texts it is common for multiple
words to be related and co-occurring in the same context. We extend the notion
of co-occurrences to cover $k(\geq\!\!2)$-way co-occurrences among a set of
$k$-words. Specifically, we prove a theoretical relationship between the joint
probability of $k(\geq\!\!2)$ words, and the sum of $\ell_2$ norms of their
embeddings. Next, we propose a learning objective motivated by our theoretical
result that utilises $k$-way co-occurrences for learning word embeddings. Our
experimental results show that the derived theoretical relationship does indeed
hold empirically, and despite data sparsity, for some smaller $k$ values,
$k$-way embeddings perform comparably or better than $2$-way embeddings in a
range of tasks.
