  Availability of labelled data is the major obstacle to the deployment of deep
learning algorithms for computer vision tasks in new domains. The fact that
many frameworks adopted to solve different tasks share the same architecture
suggests that there should be a way of reusing the knowledge learned in a
specific setting to solve novel tasks with limited or no additional
supervision. In this work, we first show that such knowledge can be shared
across tasks by learning a mapping between task-specific deep features in a
given domain. Then, we show that this mapping function, implemented by a neural
network, is able to generalize to novel unseen domains. Besides, we propose a
set of strategies to constrain the learned feature spaces, to ease learning and
increase the generalization capability of the mapping network, thereby
considerably improving the final performance of our framework. Our proposal
obtains compelling results in challenging synthetic-to-real adaptation
scenarios by transferring knowledge between monocular depth estimation and
semantic segmentation tasks.
