  Class-incremental semantic image segmentation assumes multiple model updates,
each enriching the model to segment new categories. This is typically carried
out by providing expensive pixel-level annotations to the training algorithm
for all new objects, limiting the adoption of such methods in practical
applications. Approaches that solely require image-level labels offer an
attractive alternative, yet, such coarse annotations lack precise information
about the location and boundary of the new objects. In this paper we argue
that, since classes represent not just indices but semantic entities, the
conceptual relationships between them can provide valuable information that
should be leveraged. We propose a weakly supervised approach that exploits such
semantic relations to transfer objectness prior from the previously learned
classes into the new ones, complementing the supervisory signal from
image-level labels. We validate our approach on a number of continual learning
tasks, and show how even a simple pairwise interaction between classes can
significantly improve the segmentation mask quality of both old and new
classes. We show these conclusions still hold for longer and, hence, more
realistic sequences of tasks and for a challenging few-shot scenario.
