  In this work, we investigate the inductive biases that arise from learning
multiple tasks, either simultaneously (multi-task learning, MTL) or
sequentially (pretraining and subsequent finetuning, PT+FT). We describe novel
implicit regularization penalties associated with MTL and PT+FT in diagonal
linear networks and single-hidden-layer ReLU networks. These penalties indicate
that MTL and PT+FT induce the network to reuse features in different ways. 1)
Both MTL and PT+FT exhibit biases towards feature reuse between tasks, and
towards sparsity in the set of learned features. We show a "conservation law"
that implies a direct tradeoff between these two biases. Our results also imply
that during finetuning, networks operate in a hybrid of the kernel (or "lazy")
regime and the feature-learning ("rich") regime identified in prior work. 2)
PT+FT exhibits a novel "nested feature selection" behavior not described by
either the lazy or rich regimes, which biases it to extract a sparse subset of
the features learned during pretraining. This regime is much narrower for MTL.
3) PT+FT (but not MTL) in ReLU networks benefits from features that are
correlated between the auxiliary and main task. We confirm our insights
empirically with teacher-student models. Finally, we validate our theory in
deep neural networks trained on image classification tasks, finding that they
may exhibit a nested feature selection regime. We also introduce a practical
technique -- weight rescaling following pretraining -- and provide evidence
that this method can improve finetuning performance by inducing the network to
operate in the nested feature selection regime.
