  In this paper, we propose a novel Pattern-Affinitive Propagation (PAP)
framework to jointly predict depth, surface normal and semantic segmentation.
The motivation behind it comes from the statistic observation that
pattern-affinitive pairs recur much frequently across different tasks as well
as within a task. Thus, we can conduct two types of propagations, cross-task
propagation and task-specific propagation, to adaptively diffuse those similar
patterns. The former integrates cross-task affinity patterns to adapt to each
task therein through the calculation on non-local relationships. Next the
latter performs an iterative diffusion in the feature space so that the
cross-task affinity patterns can be widely-spread within the task. Accordingly,
the learning of each task can be regularized and boosted by the complementary
task-level affinities. Extensive experiments demonstrate the effectiveness and
the superiority of our method on the joint three tasks. Meanwhile, we achieve
the state-of-the-art or competitive results on the three related datasets,
NYUD-v2, SUN-RGBD and KITTI.
