  Current self-supervised learning (SSL) methods (e.g., SimCLR, DINO,
VICReg,MOCOv3) target primarily on representations at instance level and do not
generalize well to dense prediction tasks, such as object detection and
segmentation.Towards aligning SSL with dense predictions, this paper
demonstrates for the first time the underlying mean-shift clustering process of
Vision Transformers (ViT), which aligns well with natural image semantics
(e.g., a world of objects and stuffs). By employing transformer for joint
embedding and clustering, we propose a two-level feature clustering SSL method,
coined Feature-Level Self-supervised Learning (FLSL). We present the formal
definition of the FLSL problem and construct the objectives from the mean-shift
and k-means perspectives. We show that FLSL promotes remarkable semantic
cluster representations and learns an embedding scheme amenable to intra-view
and inter-view feature clustering. Experiments show that FLSL yields
significant improvements in dense prediction tasks, achieving 44.9 (+2.8)% AP
and 46.5% AP in object detection, as well as 40.8 (+2.3)% AP and 42.1% AP in
instance segmentation on MS-COCO, using Mask R-CNN with ViT-S/16 and ViT-S/8 as
backbone, respectively. FLSL consistently outperforms existing SSL methods
across additional benchmarks, including UAV17 object detection on UAVDT, and
video instance segmentation on DAVIS 2017.We conclude by presenting
visualization and various ablation studies to better understand the success of
FLSL. The source code is available at https://github.com/ISL-CV/FLSL.
