  Empowered by large datasets, e.g., ImageNet, unsupervised learning on
large-scale data has enabled significant advances for classification tasks.
However, whether the large-scale unsupervised semantic segmentation can be
achieved remains unknown. There are two major challenges: i) we need a
large-scale benchmark for assessing algorithms; ii) we need to develop methods
to simultaneously learn category and shape representation in an unsupervised
manner. In this work, we propose a new problem of large-scale unsupervised
semantic segmentation (LUSS) with a newly created benchmark dataset to help the
research progress. Building on the ImageNet dataset, we propose the ImageNet-S
dataset with 1.2 million training images and 50k high-quality semantic
segmentation annotations for evaluation. Our benchmark has a high data
diversity and a clear task objective. We also present a simple yet effective
method that works surprisingly well for LUSS. In addition, we benchmark related
un/weakly/fully supervised methods accordingly, identifying the challenges and
possible directions of LUSS. The benchmark and source code is publicly
available at https://github.com/LUSSeg.
