  Semantic segmentation requires a detailed labeling of image pixels by object
category. Information derived from local image patches is necessary to describe
the detailed shape of individual objects. However, this information is
ambiguous and can result in noisy labels. Global inference of image content can
instead capture the general semantic concepts present. We advocate that
high-recall holistic inference of image concepts provides valuable information
for detailed pixel labeling. We build a two-stream neural network architecture
that facilitates information flow from holistic information to local pixels,
while keeping common image features shared among the low-level layers of both
the holistic analysis and segmentation branches. We empirically evaluate our
network on four standard semantic segmentation datasets. Our network obtains
state-of-the-art performance on PASCAL-Context and NYUDv2, and ablation studies
verify its effectiveness on ADE20K and SIFT-Flow.
