  Auxiliary-Task Learning (ATL) aims to improve the performance of the target
task by leveraging the knowledge obtained from related tasks. Occasionally,
learning multiple tasks simultaneously results in lower accuracy than learning
only the target task, which is known as negative transfer. This problem is
often attributed to the gradient conflicts among tasks, and is frequently
tackled by coordinating the task gradients in previous works. However, these
optimization-based methods largely overlook the auxiliary-target generalization
capability. To better understand the root cause of negative transfer, we
experimentally investigate it from both optimization and generalization
perspectives. Based on our findings, we introduce ForkMerge, a novel approach
that periodically forks the model into multiple branches, automatically
searches the varying task weights by minimizing target validation errors, and
dynamically merges all branches to filter out detrimental task-parameter
updates. On a series of auxiliary-task learning benchmarks, ForkMerge
outperforms existing methods and effectively mitigates negative transfer.
