  Multi-task learning (MTL) jointly learns a set of tasks by sharing parameters
among tasks. It is a promising approach for reducing storage costs while
improving task accuracy for many computer vision tasks. The effective adoption
of MTL faces two main challenges. The first challenge is to determine what
parameters to share across tasks to optimize for both memory efficiency and
task accuracy. The second challenge is to automatically apply MTL algorithms to
an arbitrary CNN backbone without requiring time-consuming manual
re-implementation and significant domain expertise. This paper addresses the
challenges by developing the first programming framework AutoMTL that automates
efficient MTL model development for vision tasks. AutoMTL takes as inputs an
arbitrary backbone convolutional neural network (CNN) and a set of tasks to
learn, and automatically produces a multi-task model that achieves high
accuracy and small memory footprint simultaneously. Experiments on three
popular MTL benchmarks (CityScapes, NYUv2, Tiny-Taskonomy) demonstrate the
effectiveness of AutoMTL over state-of-the-art approaches as well as the
generalizability of AutoMTL across CNNs. AutoMTL is open-sourced and available
at https://github.com/zhanglijun95/AutoMTL.
