  Multi-task learning (MTL) encapsulates multiple learned tasks in a single
model and often lets those tasks learn better jointly. However, when deploying
MTL onto those real-world systems that are often resource-constrained or
latency-sensitive, two prominent challenges arise: (i) during training,
simultaneously optimizing all tasks is often difficult due to gradient
conflicts across tasks; (ii) at inference, current MTL regimes have to activate
nearly the entire model even to just execute a single task. Yet most real
systems demand only one or two tasks at each moment, and switch between tasks
as needed: therefore such all tasks activated inference is also highly
inefficient and non-scalable. In this paper, we present a model-accelerator
co-design framework to enable efficient on-device MTL. Our framework, dubbed
M$^3$ViT, customizes mixture-of-experts (MoE) layers into a vision transformer
(ViT) backbone for MTL, and sparsely activates task-specific experts during
training. Then at inference with any task of interest, the same design allows
for activating only the task-corresponding sparse expert pathway, instead of
the full model. Our new model design is further enhanced by hardware-level
innovations, in particular, a novel computation reordering scheme tailored for
memory-constrained MTL that achieves zero-overhead switching between tasks and
can scale to any number of experts. When executing single-task inference,
M$^{3}$ViT achieves higher accuracies than encoder-focused MTL methods, while
significantly reducing 88% inference FLOPs. When implemented on a hardware
platform of one Xilinx ZCU104 FPGA, our co-design framework reduces the memory
requirement by 2.4 times, while achieving energy efficiency up to 9.23 times
higher than a comparable FPGA baseline. Code is available at:
https://github.com/VITA-Group/M3ViT.
