  Depth estimation and scene parsing are two particularly important tasks in
visual scene understanding. In this paper we tackle the problem of simultaneous
depth estimation and scene parsing in a joint CNN. The task can be typically
treated as a deep multi-task learning problem [42]. Different from previous
methods directly optimizing multiple tasks given the input training data, this
paper proposes a novel multi-task guided prediction-and-distillation network
(PAD-Net), which first predicts a set of intermediate auxiliary tasks ranging
from low level to high level, and then the predictions from these intermediate
auxiliary tasks are utilized as multi-modal input via our proposed multi-modal
distillation modules for the final tasks. During the joint learning, the
intermediate tasks not only act as supervision for learning more robust deep
representations but also provide rich multi-modal information for improving the
final tasks. Extensive experiments are conducted on two challenging datasets
(i.e. NYUD-v2 and Cityscapes) for both the depth estimation and scene parsing
tasks, demonstrating the effectiveness of the proposed approach.
