  Almost all existing deep learning approaches for semantic segmentation tackle
this task as a pixel-wise classification problem. Yet humans understand a scene
not in terms of pixels, but by decomposing it into perceptual groups and
structures that are the basic building blocks of recognition. This motivates us
to propose an end-to-end pixel-wise metric learning approach that mimics this
process. In our approach, the optimal visual representation determines the
right segmentation within individual images and associates segments with the
same semantic classes across images. The core visual learning problem is
therefore to maximize the similarity within segments and minimize the
similarity between segments. Given a model trained this way, inference is
performed consistently by extracting pixel-wise embeddings and clustering, with
the semantic label determined by the majority vote of its nearest neighbors
from an annotated set.
  As a result, we present the SegSort, as a first attempt using deep learning
for unsupervised semantic segmentation, achieving $76\%$ performance of its
supervised counterpart. When supervision is available, SegSort shows consistent
improvements over conventional approaches based on pixel-wise softmax training.
Additionally, our approach produces more precise boundaries and consistent
region predictions. The proposed SegSort further produces an interpretable
result, as each choice of label can be easily understood from the retrieved
nearest segments.
