  We aim to address Multi-Task Learning (MTL) with a large number of tasks by
Multi-Task Grouping (MTG). Given N tasks, we propose to simultaneously identify
the best task groups from 2^N candidates and train the model weights
simultaneously in one-shot, with the high-order task-affinity fully exploited.
This is distinct from the pioneering methods which sequentially identify the
groups and train the model weights, where the group identification often relies
on heuristics. As a result, our method not only improves the training
efficiency, but also mitigates the objective bias introduced by the sequential
procedures that potentially lead to a suboptimal solution. Specifically, we
formulate MTG as a fully differentiable pruning problem on an adaptive network
architecture determined by an underlying Categorical distribution. To
categorize N tasks into K groups (represented by K encoder branches), we
initially set up KN task heads, where each branch connects to all N task heads
to exploit the high-order task-affinity. Then, we gradually prune the KN heads
down to N by learning a relaxed differentiable Categorical distribution,
ensuring that each task is exclusively and uniquely categorized into only one
branch. Extensive experiments on CelebA and Taskonomy datasets with detailed
ablations show the promising performance and efficiency of our method. The
codes are available at https://github.com/ethanygao/DMTG.
