  Learning on text-attributed graphs (TAGs), in which nodes are associated with
one or more texts, has been the subject of much recent work. However, most
approaches tend to make strong assumptions about the downstream task of
interest, are reliant on hand-labeled data, or fail to equally balance the
importance of both text and graph representations. In this work, we propose
Contrastive Graph-Text pretraining (ConGraT), a general, self-supervised
approach for jointly learning separate representations of texts and nodes in a
TAG. Our method trains a language model (LM) and a graph neural network (GNN)
to align their representations in a common latent space using a batch-wise
contrastive learning objective inspired by CLIP. We further propose an
extension to the CLIP objective that leverages graph structure to incorporate
information about inter-node similarity. Extensive experiments demonstrate that
ConGraT outperforms baselines on various downstream tasks, including node and
text category classification, link prediction, and language modeling. Finally,
we present an application of our method to community detection in social
graphs, which enables finding more textually grounded communities, rather than
purely graph-based ones. Code and certain datasets are available at
https://github.com/wwbrannon/congrat.
