  Existing works on semantic segmentation typically consider a small number of
labels, ranging from tens to a few hundreds. With a large number of labels,
training and evaluation of such task become extremely challenging due to
correlation between labels and lack of datasets with complete annotations. We
formulate semantic segmentation as a problem of image segmentation given a
semantic concept, and propose a novel system which can potentially handle an
unlimited number of concepts, including objects, parts, stuff, and attributes.
We achieve this using a weakly and semi-supervised framework leveraging
multiple datasets with different levels of supervision. We first train a deep
neural network on a 6M stock image dataset with only image-level labels to
learn visual-semantic embedding on 18K concepts. Then, we refine and extend the
embedding network to predict an attention map, using a curated dataset with
bounding box annotations on 750 concepts. Finally, we train an attention-driven
class agnostic segmentation network using an 80-category fully annotated
dataset. We perform extensive experiments to validate that the proposed system
performs competitively to the state of the art on fully supervised concepts,
and is capable of producing accurate segmentations for weakly learned and
unseen concepts.
