  Multi-task learning (MTL) has achieved great success in various research
domains, such as CV, NLP and IR etc. Due to the complex and competing task
correlation, naive training all tasks may lead to inequitable learning, i.e.
some tasks are learned well while others are overlooked. Multi-task
optimization (MTO) aims to improve all tasks at same time, but conventional
methods often perform poor when tasks with large loss scale or gradient norm
magnitude difference. To solve the issue, we in-depth investigate the equity
problem for MTL and find that regularizing relative contribution of different
tasks (i.e. value of task-specific loss divides its raw gradient norm) in
updating shared parameter can improve generalization performance of MTL. Based
on our theoretical analysis, we propose a novel multi-task optimization method,
named EMTL, to achieve equitable MTL. Specifically, we efficiently add variance
regularization to make different tasks' relative contribution closer. Extensive
experiments have been conduct to evaluate EMTL, our method stably outperforms
state-of-the-art methods on the public benchmark datasets of two different
research domains. Furthermore, offline and online A/B test on multi-task
recommendation are conducted too. EMTL improves multi-task recommendation
significantly, demonstrating the superiority and practicability of our method
in industrial landscape.
