  Unsupervised feature learning has made great strides with contrastive
learning based on instance discrimination and invariant mapping, as benchmarked
on curated class-balanced datasets. However, natural data could be highly
correlated and long-tail distributed. Natural between-instance similarity
conflicts with the presumed instance distinction, causing unstable training and
poor performance.
  Our idea is to discover and integrate between-instance similarity into
contrastive learning, not directly by instance grouping, but by cross-level
discrimination (CLD) between instances and local instance groups. While
invariant mapping of each instance is imposed by attraction within its
augmented views, between-instance similarity could emerge from common repulsion
against instance groups.
  Our batch-wise and cross-view comparisons also greatly improve the
positive/negative sample ratio of contrastive learning and achieve better
invariant mapping. To effect both grouping and discrimination objectives, we
impose them on features separately derived from a shared representation. In
addition, we propose normalized projection heads and unsupervised
hyper-parameter tuning for the first time.
  Our extensive experimentation demonstrates that CLD is a lean and powerful
add-on to existing methods such as NPID, MoCo, InfoMin, and BYOL on highly
correlated, long-tail, or balanced datasets. It not only achieves new
state-of-the-art on self-supervision, semi-supervision, and transfer learning
benchmarks, but also beats MoCo v2 and SimCLR on every reported performance
attained with a much larger compute. CLD effectively brings unsupervised
learning closer to natural data and real-world applications. Our code is
publicly available at: https://github.com/frank-xwang/CLD-UnsupervisedLearning.
