  Removing supervision in semantic segmentation is still tricky. Current
approaches can deal with common categorical patterns yet resort to multi-stage
architectures. We design a novel end-to-end model leveraging local-global patch
matching to predict categories, good localization, area and shape of objects
for semantic segmentation. The local-global matching is, in turn, compelled by
optimal transport plans fulfilling area constraints nearing a solution for
exact shape prediction. Our model attains state-of-the-art in Weakly Supervised
Semantic Segmentation, only image-level labels, with 75% mIoU on PascalVOC2012
val set and 46% on MS-COCO2014 val set. Dropping the image-level labels and
clustering self-supervised learned features to yield pseudo-multi-level labels,
we obtain an unsupervised model for semantic segmentation. We also attain
state-of-the-art on Unsupervised Semantic Segmentation with 43.6% mIoU on
PascalVOC2012 val set and 19.4% on MS-COCO2014 val set. The code is available
at https://github.com/deepplants/PC2M.
