  Vulnerability to adversarial attacks is a well-known weakness of Deep Neural
networks. While most of the studies focus on single-task neural networks with
computer vision datasets, very little research has considered complex
multi-task models that are common in real applications. In this paper, we
evaluate the design choices that impact the robustness of multi-task deep
learning networks. We provide evidence that blindly adding auxiliary tasks, or
weighing the tasks provides a false sense of robustness. Thereby, we tone down
the claim made by previous research and study the different factors which may
affect robustness. In particular, we show that the choice of the task to
incorporate in the loss function are important factors that can be leveraged to
yield more robust models.
