  CNNs and Transformers have their own advantages and both have been widely
used for dense prediction in multi-task learning (MTL). Most of the current
studies on MTL solely rely on CNN or Transformer. In this work, we present a
novel MTL model by combining both merits of deformable CNN and query-based
Transformer with shared gating for multi-task learning of dense prediction.
This combination may offer a simple and efficient solution owing to its
powerful and flexible task-specific learning and advantages of lower cost, less
complexity and smaller parameters than the traditional MTL methods. We
introduce deformable mixer Transformer with gating (DeMTG), a simple and
effective encoder-decoder architecture up-to-date that incorporates the
convolution and attention mechanism in a unified network for MTL. It is
exquisitely designed to use advantages of each block, and provide deformable
and comprehensive features for all tasks from local and global perspective.
First, the deformable mixer encoder contains two types of operators: the
channel-aware mixing operator leveraged to allow communication among different
channels, and the spatial-aware deformable operator with deformable convolution
applied to efficiently sample more informative spatial locations. Second, the
task-aware gating transformer decoder is used to perform the task-specific
predictions, in which task interaction block integrated with self-attention is
applied to capture task interaction features, and the task query block
integrated with gating attention is leveraged to select corresponding
task-specific features. Further, the experiment results demonstrate that the
proposed DeMTG uses fewer GFLOPs and significantly outperforms current
Transformer-based and CNN-based competitive models on a variety of metrics on
three dense prediction datasets. Our code and models are available at
https://github.com/yangyangxu0/DeMTG.
