  Although well-known large-scale datasets, such as ImageNet, have driven image
understanding forward, most of these datasets require extensive manual
annotation and are thus not easily scalable. This limits the advancement of
image understanding techniques. The impact of these large-scale datasets can be
observed in almost every vision task and technique in the form of pre-training
for initialization. In this work, we propose an easily scalable and
self-supervised technique that can be used to pre-train any semantic RGB
segmentation method. In particular, our pre-training approach makes use of
automatically generated labels that can be obtained using depth sensors. These
labels, denoted by HN-labels, represent different height and normal patches,
which allow mining of local semantic information that is useful in the task of
semantic RGB segmentation. We show how our proposed self-supervised
pre-training with HN-labels can be used to replace ImageNet pre-training, while
using 25x less images and without requiring any manual labeling. We pre-train a
semantic segmentation network with our HN-labels, which resembles our final
task more than pre-training on a less related task, e.g. classification with
ImageNet. We evaluate on two datasets (NYUv2 and CamVid), and we show how the
similarity in tasks is advantageous not only in speeding up the pre-training
process, but also in achieving better final semantic segmentation accuracy than
ImageNet pre-training
