  Multi-task learning promises better model generalization on a target task by
jointly optimizing it with an auxiliary task. However, the current practice
requires additional labeling efforts for the auxiliary task, while not
guaranteeing better model performance. In this paper, we find that jointly
training a dense prediction (target) task with a self-supervised (auxiliary)
task can consistently improve the performance of the target task, while
eliminating the need for labeling auxiliary tasks. We refer to this joint
training as Composite Learning (CompL). Experiments of CompL on monocular depth
estimation, semantic segmentation, and boundary detection show consistent
performance improvements in fully and partially labeled datasets. Further
analysis on depth estimation reveals that joint training with self-supervision
outperforms most labeled auxiliary tasks. We also find that CompL can improve
model robustness when the models are evaluated in new domains. These results
demonstrate the benefits of self-supervision as an auxiliary task, and
establish the design of novel task-specific self-supervised methods as a new
axis of investigation for future multi-task learning research.
