  Multi-task networks are commonly utilized to alleviate the need for a large
number of highly specialized single-task networks. However, two common
challenges in developing multi-task models are often overlooked in literature.
First, enabling the model to be inherently incremental, continuously
incorporating information from new tasks without forgetting the previously
learned ones (incremental learning). Second, eliminating adverse interactions
amongst tasks, which has been shown to significantly degrade the single-task
performance in a multi-task setup (task interference). In this paper, we show
that both can be achieved simply by reparameterizing the convolutions of
standard neural network architectures into a non-trainable shared part (filter
bank) and task-specific parts (modulators), where each modulator has a fraction
of the filter bank parameters. Thus, our reparameterization enables the model
to learn new tasks without adversely affecting the performance of existing
ones. The results of our ablation study attest the efficacy of the proposed
reparameterization. Moreover, our method achieves state-of-the-art on two
challenging multi-task learning benchmarks, PASCAL-Context and NYUD, and also
demonstrates superior incremental learning capability as compared to its close
competitors.
