  Semantic segmentation has achieved remarkable progress but remains
challenging due to the complex scene, object occlusion, and so on. Some
research works have attempted to use extra information such as a depth map to
help RGB based semantic segmentation because the depth map could provide
complementary geometric cues. However, due to the inaccessibility of depth
sensors, depth information is usually unavailable for the test images. In this
paper, we leverage only the depth of training images as the privileged
information to mine the hard pixels in semantic segmentation, in which depth
information is only available for training images but not available for test
images. Specifically, we propose a novel Loss Weight Module, which outputs a
loss weight map by employing two depth-related measurements of hard pixels:
Depth Prediction Error and Depthaware Segmentation Error. The loss weight map
is then applied to segmentation loss, with the goal of learning a more robust
model by paying more attention to the hard pixels. Besides, we also explore a
curriculum learning strategy based on the loss weight map. Meanwhile, to fully
mine the hard pixels on different scales, we apply our loss weight module to
multi-scale side outputs. Our hard pixels mining method achieves the
state-of-the-art results on two benchmark datasets, and even outperforms the
methods which need depth input during testing.
