  Humans are able to recognize structured relations in observation, allowing us
to decompose complex scenes into simpler parts and abstract the visual world in
multiple levels. However, such hierarchical reasoning ability of human
perception remains largely unexplored in current literature of semantic
segmentation. Existing work is often aware of flatten labels and predicts
target classes exclusively for each pixel. In this paper, we instead address
hierarchical semantic segmentation (HSS), which aims at structured, pixel-wise
description of visual observation in terms of a class hierarchy. We devise
HSSN, a general HSS framework that tackles two critical issues in this task: i)
how to efficiently adapt existing hierarchy-agnostic segmentation networks to
the HSS setting, and ii) how to leverage the hierarchy information to
regularize HSS network learning. To address i), HSSN directly casts HSS as a
pixel-wise multi-label classification task, only bringing minimal architecture
change to current segmentation models. To solve ii), HSSN first explores
inherent properties of the hierarchy as a training objective, which enforces
segmentation predictions to obey the hierarchy structure. Further, with
hierarchy-induced margin constraints, HSSN reshapes the pixel embedding space,
so as to generate well-structured pixel representations and improve
segmentation eventually. We conduct experiments on four semantic segmentation
datasets (i.e., Mapillary Vistas 2.0, Cityscapes, LIP, and PASCAL-Person-Part),
with different class hierarchies, segmentation network architectures and
backbones, showing the generalization and superiority of HSSN.
