  In this paper, we present and study a new image segmentation task, called
Generalized Open-set Semantic Segmentation (GOSS). Previously, with the
well-known open-set semantic segmentation (OSS), the intelligent agent only
detects the unknown regions without further processing, limiting their
perception of the environment. It stands to reason that a further analysis of
the detected unknown pixels would be beneficial. Therefore, we propose GOSS,
which unifies the abilities of two well-defined segmentation tasks, OSS and
generic segmentation (GS), in a holistic way. Specifically, GOSS classifies
pixels as belonging to known classes, and clusters (or groups) of pixels of
unknown class are labelled as such. To evaluate this new expanded task, we
further propose a metric which balances the pixel classification and clustering
aspects. Moreover, we build benchmark tests on top of existing datasets and
propose a simple neural architecture as a baseline, which jointly predicts
pixel classification and clustering under open-set settings. Our experiments on
multiple benchmarks demonstrate the effectiveness of our baseline. We believe
our new GOSS task can produce an expressive image understanding for future
research. Code will be made available.
