  Multi-task learning solves multiple correlated tasks. However, conflicts may
exist between them. In such circumstances, a single solution can rarely
optimize all the tasks, leading to performance trade-offs. To arrive at a set
of optimized yet well-distributed models that collectively embody different
trade-offs in one algorithmic pass, this paper proposes to view Pareto
multi-task learning through the lens of multi-task optimization. Multi-task
learning is first cast as a multi-objective optimization problem, which is then
decomposed into a diverse set of unconstrained scalar-valued subproblems. These
subproblems are solved jointly using a novel multi-task gradient descent
method, whose uniqueness lies in the iterative transfer of model parameters
among the subproblems during the course of optimization. A theorem proving
faster convergence through the inclusion of such transfers is presented. We
investigate the proposed multi-task learning with multi-task optimization for
solving various problem settings including image classification, scene
understanding, and multi-target regression. Comprehensive experiments confirm
that the proposed method significantly advances the state-of-the-art in
discovering sets of Pareto-optimized models. Notably, on the large image
dataset we tested on, namely NYUv2, the hypervolume convergence achieved by our
method was found to be nearly two times faster than the next-best among the
state-of-the-art.
