  We propose a novel multi-task learning architecture, which allows learning of
task-specific feature-level attention. Our design, the Multi-Task Attention
Network (MTAN), consists of a single shared network containing a global feature
pool, together with a soft-attention module for each task. These modules allow
for learning of task-specific features from the global features, whilst
simultaneously allowing for features to be shared across different tasks. The
architecture can be trained end-to-end and can be built upon any feed-forward
neural network, is simple to implement, and is parameter efficient. We evaluate
our approach on a variety of datasets, across both image-to-image predictions
and image classification tasks. We show that our architecture is
state-of-the-art in multi-task learning compared to existing methods, and is
also less sensitive to various weighting schemes in the multi-task loss
function. Code is available at https://github.com/lorenmt/mtan.
