  Hierarchical semantic structures naturally exist in an image dataset, in
which several semantically relevant image clusters can be further integrated
into a larger cluster with coarser-grained semantics. Capturing such structures
with image representations can greatly benefit the semantic understanding on
various downstream tasks. Existing contrastive representation learning methods
lack such an important model capability. In addition, the negative pairs used
in these methods are not guaranteed to be semantically distinct, which could
further hamper the structural correctness of learned image representations. To
tackle these limitations, we propose a novel contrastive learning framework
called Hierarchical Contrastive Selective Coding (HCSC). In this framework, a
set of hierarchical prototypes are constructed and also dynamically updated to
represent the hierarchical semantic structures underlying the data in the
latent space. To make image representations better fit such semantic
structures, we employ and further improve conventional instance-wise and
prototypical contrastive learning via an elaborate pair selection scheme. This
scheme seeks to select more diverse positive pairs with similar semantics and
more precise negative pairs with truly distinct semantics. On extensive
downstream tasks, we verify the superior performance of HCSC over
state-of-the-art contrastive methods, and the effectiveness of major model
components is proved by plentiful analytical studies. We build a comprehensive
model zoo in Sec. D. Our source code and model weights are available at
https://github.com/gyfastas/HCSC
