  To date, most existing self-supervised learning methods are designed and
optimized for image classification. These pre-trained models can be sub-optimal
for dense prediction tasks due to the discrepancy between image-level
prediction and pixel-level prediction. To fill this gap, we aim to design an
effective, dense self-supervised learning method that directly works at the
level of pixels (or local features) by taking into account the correspondence
between local features. We present dense contrastive learning, which implements
self-supervised learning by optimizing a pairwise contrastive (dis)similarity
loss at the pixel level between two views of input images. Compared to the
baseline method MoCo-v2, our method introduces negligible computation overhead
(only <1% slower), but demonstrates consistently superior performance when
transferring to downstream dense prediction tasks including object detection,
semantic segmentation and instance segmentation; and outperforms the
state-of-the-art methods by a large margin. Specifically, over the strong
MoCo-v2 baseline, our method achieves significant improvements of 2.0% AP on
PASCAL VOC object detection, 1.1% AP on COCO object detection, 0.9% AP on COCO
instance segmentation, 3.0% mIoU on PASCAL VOC semantic segmentation and 1.8%
mIoU on Cityscapes semantic segmentation. Code is available at:
https://git.io/AdelaiDet
