  The goal of multi-task learning is to learn diverse tasks within a single
unified network. As each task has its own unique objective function, conflicts
emerge during training, resulting in negative transfer among them. Earlier
research identified these conflicting gradients in shared parameters between
tasks and attempted to realign them in the same direction. However, we prove
that such optimization strategies lead to sub-optimal Pareto solutions due to
their inability to accurately determine the individual contributions of each
parameter across various tasks. In this paper, we propose the concept of task
priority to evaluate parameter contributions across different tasks. To learn
task priority, we identify the type of connections related to links between
parameters influenced by task-specific losses during backpropagation. The
strength of connections is gauged by the magnitude of parameters to determine
task priority. Based on these, we present a new method named connection
strength-based optimization for multi-task learning which consists of two
phases. The first phase learns the task priority within the network, while the
second phase modifies the gradients while upholding this priority. This
ultimately leads to finding new Pareto optimal solutions for multiple tasks.
Through extensive experiments, we show that our approach greatly enhances
multi-task performance in comparison to earlier gradient manipulation methods.
