  Recent work has found that multi-task training with a large number of diverse
tasks can uniformly improve downstream performance on unseen target tasks. In
contrast, literature on task transferability has established that the choice of
intermediate tasks can heavily affect downstream task performance. In this
work, we aim to disentangle the effect of scale and relatedness of tasks in
multi-task representation learning. We find that, on average, increasing the
scale of multi-task learning, in terms of the number of tasks, indeed results
in better learned representations than smaller multi-task setups. However, if
the target tasks are known ahead of time, then training on a smaller set of
related tasks is competitive to the large-scale multi-task training at a
reduced computational cost.
