  Panoptic segmentation methods assign a known class to each pixel given in
input. Even for state-of-the-art approaches, this inevitably enforces decisions
that systematically lead to wrong predictions for objects outside the training
categories. However, robustness against out-of-distribution samples and corner
cases is crucial in safety-critical settings to avoid dangerous consequences.
Since real-world datasets cannot contain enough data points to adequately
sample the long tail of the underlying distribution, models must be able to
deal with unseen and unknown scenarios as well. Previous methods targeted this
by re-identifying already-seen unlabeled objects. In this work, we propose the
necessary step to extend segmentation with a new setting which we term holistic
segmentation. Holistic segmentation aims to identify and separate objects of
unseen, unknown categories into instances without any prior knowledge about
them while performing panoptic segmentation of known classes. We tackle this
new problem with U3HS, which finds unknowns as highly uncertain regions and
clusters their corresponding instance-aware embeddings into individual objects.
By doing so, for the first time in panoptic segmentation with unknown objects,
our U3HS is trained without unknown categories, reducing assumptions and
leaving the settings as unconstrained as in real-life scenarios. Extensive
experiments on public data from MS COCO, Cityscapes, and Lost&Found demonstrate
the effectiveness of U3HS for this new, challenging, and assumptions-free
setting called holistic segmentation. Project page:
https://holisticseg.github.io.
