  Self-supervised learning (SSL) with vision transformers (ViTs) has proven
effective for representation learning as demonstrated by the impressive
performance on various downstream tasks. Despite these successes, existing
ViT-based SSL architectures do not fully exploit the ViT backbone, particularly
the patch tokens of the ViT. In this paper, we introduce a novel Semantic Graph
Consistency (SGC) module to regularize ViT-based SSL methods and leverage patch
tokens effectively. We reconceptualize images as graphs, with image patches as
nodes and infuse relational inductive biases by explicit message passing using
Graph Neural Networks into the SSL framework. Our SGC loss acts as a
regularizer, leveraging the underexploited patch tokens of ViTs to construct a
graph and enforcing consistency between graph features across multiple views of
an image. Extensive experiments on various datasets including ImageNet, RESISC
and Food-101 show that our approach significantly improves the quality of
learned representations, resulting in a 5-10\% increase in performance when
limited labeled data is used for linear evaluation. These experiments coupled
with a comprehensive set of ablations demonstrate the promise of our approach
in various settings.
