  Incorporating heterogeneous representations from different architectures has
facilitated various vision tasks, e.g., some hybrid networks combine
transformers and convolutions. However, complementarity between such
heterogeneous architectures has not been well exploited in self-supervised
learning. Thus, we propose Heterogeneous Self-Supervised Learning (HSSL), which
enforces a base model to learn from an auxiliary head whose architecture is
heterogeneous from the base model. In this process, HSSL endows the base model
with new characteristics in a representation learning way without structural
changes. To comprehensively understand the HSSL, we conduct experiments on
various heterogeneous pairs containing a base model and an auxiliary head. We
discover that the representation quality of the base model moves up as their
architecture discrepancy grows. This observation motivates us to propose a
search strategy that quickly determines the most suitable auxiliary head for a
specific base model to learn and several simple but effective methods to
enlarge the model discrepancy. The HSSL is compatible with various
self-supervised methods, achieving superior performances on various downstream
tasks, including image classification, semantic segmentation, instance
segmentation, and object detection. Our source code will be made publicly
available.
