  Multi-Task Learning (MTL) involves developing a singular model, known as a
multi-task model, to concurrently perform multiple tasks. While the security of
single-task models has been thoroughly studied, multi-task models pose several
critical security questions, such as 1) their vulnerability to single-task
adversarial attacks, 2) the possibility of designing attacks that target
multiple tasks, and 3) the impact of task sharing and adversarial training on
their resilience to such attacks. This paper addresses these queries through
detailed analysis and rigorous experimentation. First, we explore the
adaptation of single-task white-box attacks to multi-task models and identify
their limitations. We then introduce a novel attack framework, the Gradient
Balancing Multi-Task Attack (GB-MTA), which treats attacking a multi-task model
as an optimization problem. This problem, based on averaged relative loss
change across tasks, is approximated as an integer linear programming problem.
Extensive evaluations on MTL benchmarks, NYUv2 and Tiny-Taxonomy, demonstrate
GB-MTA's effectiveness against both standard and adversarially trained
multi-task models. The results also highlight a trade-off between task accuracy
improvement via parameter sharing and increased model vulnerability due to
enhanced attack transferability.
