  Panoptic segmentation brings together two separate tasks: instance and
semantic segmentation. Although they are related, unifying them faces an
apparent paradox: how to learn simultaneously instance-specific and
category-specific (i.e. instance-agnostic) representations jointly. Hence,
state-of-the-art panoptic segmentation methods use complex models with a
distinct stream for each task. In contrast, we propose Hierarchical Lov\'asz
Embeddings, per pixel feature vectors that simultaneously encode instance- and
category-level discriminative information. We use a hierarchical Lov\'asz hinge
loss to learn a low-dimensional embedding space structured into a unified
semantic and instance hierarchy without requiring separate network branches or
object proposals. Besides modeling instances precisely in a proposal-free
manner, our Hierarchical Lov\'asz Embeddings generalize to categories by using
a simple Nearest-Class-Mean classifier, including for non-instance "stuff"
classes where instance segmentation methods are not applicable. Our simple
model achieves state-of-the-art results compared to existing proposal-free
panoptic segmentation methods on Cityscapes, COCO, and Mapillary Vistas.
Furthermore, our model demonstrates temporal stability between video frames.
