  A wide range of graph embedding objectives decompose into two components: one
that attracts the embeddings of nodes that are perceived as similar, and
another that repels embeddings of nodes that are perceived as dissimilar.
Because real-world graphs are sparse and the number of dissimilar pairs grows
quadratically with the number of nodes, Skip-Gram Negative Sampling (SGNS) has
emerged as a popular and efficient repulsion approach. SGNS repels each node
from a sample of dissimilar nodes, as opposed to all dissimilar nodes. In this
work, we show that node-wise repulsion is, in aggregate, an approximate
re-centering of the node embedding dimensions. Such dimension operations are
much more scalable than node operations. The dimension approach, in addition to
being more efficient, yields a simpler geometric interpretation of the
repulsion. Our result extends findings from the self-supervised learning
literature to the skip-gram model, establishing a connection between skip-gram
node contrast and dimension regularization. We show that in the limit of large
graphs, under mild regularity conditions, the original node repulsion objective
converges to optimization with dimension regularization. We use this
observation to propose an algorithm augmentation framework that speeds up any
existing algorithm, supervised or unsupervised, using SGNS. The framework
prioritizes node attraction and replaces SGNS with dimension regularization. We
instantiate this generic framework for LINE and node2vec and show that the
augmented algorithms preserve downstream performance while dramatically
increasing efficiency.
