  Unsupervised Semantic Segmentation (USS) involves segmenting images without
relying on predefined labels, aiming to alleviate the burden of extensive human
labeling. Existing methods utilize features generated by self-supervised models
and specific priors for clustering. However, their clustering objectives are
not involved in the optimization of the features during training. Additionally,
due to the lack of clear class definitions in USS, the resulting segments may
not align well with the clustering objective. In this paper, we introduce a
novel approach called Optimally Matched Hierarchy (OMH) to simultaneously
address the above issues. The core of our method lies in imposing structured
sparsity on the feature space, which allows the features to encode information
with different levels of granularity. The structure of this sparsity stems from
our hierarchy (OMH). To achieve this, we learn a soft but sparse hierarchy
among parallel clusters through Optimal Transport. Our OMH yields better
unsupervised segmentation performance compared to existing USS methods. Our
extensive experiments demonstrate the benefits of OMH when utilizing our
differentiable paradigm. We will make our code publicly available.
