  Multi-task learning (MTL) aims at solving multiple related tasks
simultaneously and has experienced rapid growth in recent years. However, MTL
models often suffer from performance degeneration with negative transfer due to
learning several tasks simultaneously. Some related work attributed the source
of the problem is the conflicting gradients. In this case, it is needed to
select useful gradient updates for all tasks carefully. To this end, we propose
a novel optimization approach for MTL, named GDOD, which manipulates gradients
of each task using an orthogonal basis decomposed from the span of all task
gradients. GDOD decomposes gradients into task-shared and task-conflict
components explicitly and adopts a general update rule for avoiding
interference across all task gradients. This allows guiding the update
directions depending on the task-shared components. Moreover, we prove the
convergence of GDOD theoretically under both convex and non-convex assumptions.
Experiment results on several multi-task datasets not only demonstrate the
significant improvement of GDOD performed to existing MTL models but also prove
that our algorithm outperforms state-of-the-art optimization methods in terms
of AUC and Logloss metrics.
