  The study of multi-task learning has drawn great attention from the
community. Despite the remarkable progress, the challenge of optimally learning
different tasks simultaneously remains to be explored. Previous works attempt
to modify the gradients from different tasks. Yet these methods give a
subjective assumption of the relationship between tasks, and the modified
gradient may be less accurate. In this paper, we introduce Stochastic Task
Allocation~(STA), a mechanism that addresses this issue by a task allocation
approach, in which each sample is randomly allocated a subset of tasks. For
further progress, we propose Interleaved Stochastic Task Allocation~(ISTA) to
iteratively allocate all tasks to each example during several consecutive
iterations. We evaluate STA and ISTA on various datasets and applications:
NYUv2, Cityscapes, and COCO for scene understanding and instance segmentation.
Our experiments show both STA and ISTA outperform current state-of-the-art
methods. The code will be available.
