  Unsupervised semantic segmentation aims to discover and localize semantically
meaningful categories within image corpora without any form of annotation. To
solve this task, algorithms must produce features for every pixel that are both
semantically meaningful and compact enough to form distinct clusters. Unlike
previous works which achieve this with a single end-to-end framework, we
propose to separate feature learning from cluster compactification.
Empirically, we show that current unsupervised feature learning frameworks
already generate dense features whose correlations are semantically consistent.
This observation motivates us to design STEGO ($\textbf{S}$elf-supervised
$\textbf{T}$ransformer with $\textbf{E}$nergy-based $\textbf{G}$raph
$\textbf{O}$ptimization), a novel framework that distills unsupervised features
into high-quality discrete semantic labels. At the core of STEGO is a novel
contrastive loss function that encourages features to form compact clusters
while preserving their relationships across the corpora. STEGO yields a
significant improvement over the prior state of the art, on both the CocoStuff
($\textbf{+14 mIoU}$) and Cityscapes ($\textbf{+9 mIoU}$) semantic segmentation
challenges.
