  Representation learning for images has been advanced by recent progress in
more complex neural models such as the Vision Transformers and new learning
theories such as the structural causal models. However, these models mainly
rely on the classification loss to implicitly regularize the class-level data
distributions, and they may face difficulties when handling classes with
diverse visual patterns. We argue that the incorporation of the structural
information between data samples may improve this situation. To achieve this
goal, this paper presents a framework termed \textbf{C}lass-level Structural
Relation Modeling and Smoothing for Visual Representation Learning (CSRMS),
which includes the Class-level Relation Modelling, Class-aware Graph Sampling,
and Relational Graph-Guided Representation Learning modules to model a
relational graph of the entire dataset and perform class-aware smoothing and
regularization operations to alleviate the issue of intra-class visual
diversity and inter-class similarity. Specifically, the Class-level Relation
Modelling module uses a clustering algorithm to learn the data distributions in
the feature space and identify three types of class-level sample relations for
the training set; Class-aware Graph Sampling module extends typical training
batch construction process with three strategies to sample dataset-level
sub-graphs; and Relational Graph-Guided Representation Learning module employs
a graph convolution network with knowledge-guided smoothing operations to ease
the projection from different visual patterns to the same class. Experiments
demonstrate the effectiveness of structured knowledge modelling for enhanced
representation learning and show that CSRMS can be incorporated with any
state-of-the-art visual representation learning models for performance gains.
The source codes and demos have been released at
https://github.com/czt117/CSRMS.
