  This paper presents CLUSTERFORMER, a universal vision model that is based on
the CLUSTERing paradigm with TransFORMER. It comprises two novel designs: 1.
recurrent cross-attention clustering, which reformulates the cross-attention
mechanism in Transformer and enables recursive updates of cluster centers to
facilitate strong representation learning; and 2. feature dispatching, which
uses the updated cluster centers to redistribute image features through
similarity-based metrics, resulting in a transparent pipeline. This elegant
design streamlines an explainable and transferable workflow, capable of
tackling heterogeneous vision tasks (i.e., image classification, object
detection, and image segmentation) with varying levels of clustering
granularity (i.e., image-, box-, and pixel-level). Empirical results
demonstrate that CLUSTERFORMER outperforms various well-known specialized
architectures, achieving 83.41% top-1 acc. over ImageNet-1K for image
classification, 54.2% and 47.0% mAP over MS COCO for object detection and
instance segmentation, 52.4% mIoU over ADE20K for semantic segmentation, and
55.8% PQ over COCO Panoptic for panoptic segmentation. For its efficacy, we
hope our work can catalyze a paradigm shift in universal models in computer
vision.
