  Multitask learning (MTL) has gained prominence for its ability to jointly
predict multiple tasks, achieving better per-task performance while using fewer
per-task model parameters than single-task learning. More recently,
decoder-focused architectures have considerably improved multitask performance
by refining task predictions using the features of other related tasks.
However, most of these refinement methods fail to simultaneously capture local
and global task-specific representations, as well as cross-task patterns in a
parameter-efficient manner. In this paper, we introduce the Efficient Multitask
Affinity Learning Network (EMA-Net), which is a lightweight framework that
enhances the task refinement capabilities of multitask networks. EMA-Net
adeptly captures local, global, and cross-task interactions using our novel
Cross-Task Affinity Learning (CTAL) module. The key innovation of CTAL lies in
its ability to manipulate task affinity matrices in a manner that is optimally
suited to apply parameter-efficient grouped convolutions without worrying about
information loss. Our results show that we achieve state-of-the-art MTL
performance for CNN-based decoder-focused models while using substantially
fewer model parameters. Our code is publicly available at
https://github.com/Armanfard-Lab/EMA-Net.
