  Multi-task learning (MTL) can improve the generalization performance of
neural networks by sharing representations with related tasks. Nonetheless, MTL
can also degrade performance through harmful interference between tasks. Recent
work has pursued task-specific loss weighting as a solution for this
interference. However, existing algorithms treat tasks as atomic, lacking the
ability to explicitly separate harmful and helpful signals beyond the task
level. To this end, we propose SLGrad, a sample-level weighting algorithm for
multi-task learning with auxiliary tasks. Through sample-specific task weights,
SLGrad reshapes the task distributions during training to eliminate harmful
auxiliary signals and augment useful task signals. Substantial generalization
performance gains are observed on (semi-) synthetic datasets and common
supervised multi-task problems.
