  Contrastive learning has recently shown immense potential in unsupervised
visual representation learning. Existing studies in this track mainly focus on
intra-image invariance learning. The learning typically uses rich intra-image
transformations to construct positive pairs and then maximizes agreement using
a contrastive loss. The merits of inter-image invariance, conversely, remain
much less explored. One major obstacle to exploit inter-image invariance is
that it is unclear how to reliably construct inter-image positive pairs, and
further derive effective supervision from them since no pair annotations are
available. In this work, we present a comprehensive empirical study to better
understand the role of inter-image invariance learning from three main
constituting components: pseudo-label maintenance, sampling strategy, and
decision boundary design. To facilitate the study, we introduce a unified and
generic framework that supports the integration of unsupervised intra- and
inter-image invariance learning. Through carefully-designed comparisons and
analysis, multiple valuable observations are revealed: 1) online labels
converge faster and perform better than offline labels; 2) semi-hard negative
samples are more reliable and unbiased than hard negative samples; 3) a less
stringent decision boundary is more favorable for inter-image invariance
learning. With all the obtained recipes, our final model, namely InterCLR,
shows consistent improvements over state-of-the-art intra-image invariance
learning methods on multiple standard benchmarks. We hope this work will
provide useful experience for devising effective unsupervised inter-image
invariance learning. Code: https://github.com/open-mmlab/mmselfsup.
