  We show that removing sigmoid transformation in the skip-gram with negative
sampling (SGNS) objective does not harm the quality of word vectors
significantly and at the same time is related to factorizing a squashed shifted
PMI matrix which, in turn, can be treated as a connection probabilities matrix
of a random graph. Empirically, such graph is a complex network, i.e. it has
strong clustering and scale-free degree distribution, and is tightly connected
with hyperbolic spaces. In short, we show the connection between static word
embeddings and hyperbolic spaces through the squashed shifted PMI matrix using
analytical and empirical methods.
