  Self-supervised learning based on instance discrimination has shown
remarkable progress. In particular, contrastive learning, which regards each
image as well as its augmentations as an individual class and tries to
distinguish them from all other images, has been verified effective for
representation learning. However, pushing away two images that are de facto
similar is suboptimal for general representation. In this paper, we propose a
hierarchical semantic alignment strategy via expanding the views generated by a
single image to \textbf{Cross-samples and Multi-level} representation, and
models the invariance to semantically similar images in a hierarchical way.
This is achieved by extending the contrastive loss to allow for multiple
positives per anchor, and explicitly pulling semantically similar
images/patches together at different layers of the network. Our method, termed
as CsMl, has the ability to integrate multi-level visual representations across
samples in a robust way. CsMl is applicable to current contrastive learning
based methods and consistently improves the performance. Notably, using the
moco as an instantiation, CsMl achieves a \textbf{76.6\% }top-1 accuracy with
linear evaluation using ResNet-50 as backbone, and \textbf{66.7\%} and
\textbf{75.1\%} top-1 accuracy with only 1\% and 10\% labels, respectively.
\textbf{All these numbers set the new state-of-the-art.}
