  Multi-task learning (MTL) models have demonstrated impressive results in
computer vision, natural language processing, and recommender systems. Even
though many approaches have been proposed, how well these approaches balance
different tasks on each parameter still remains unclear. In this paper, we
propose to measure the task dominance degree of a parameter by the total
updates of each task on this parameter. Specifically, we compute the total
updates by the exponentially decaying Average of the squared Updates (AU) on a
parameter from the corresponding task.Based on this novel metric, we observe
that many parameters in existing MTL methods, especially those in the higher
shared layers, are still dominated by one or several tasks. The dominance of AU
is mainly due to the dominance of accumulative gradients from one or several
tasks. Motivated by this, we propose a Task-wise Adaptive learning rate
approach, AdaTask in short, to separate the \emph{accumulative gradients} and
hence the learning rate of each task for each parameter in adaptive learning
rate approaches (e.g., AdaGrad, RMSProp, and Adam). Comprehensive experiments
on computer vision and recommender system MTL datasets demonstrate that AdaTask
significantly improves the performance of dominated tasks, resulting SOTA
average task-wise performance. Analysis on both synthetic and real-world
datasets shows AdaTask balance parameters in every shared layer well.
