  The fine-tuning of deep pre-trained models has recently revealed
compositional properties. This enables the arbitrary composition of multiple
specialized modules into a single, multi-task model. However, identifying the
conditions that promote compositionality remains an open issue, with recent
efforts concentrating mainly on linearized networks. We conduct a theoretical
study that attempts to demystify compositionality in standard non-linear
networks through the second-order Taylor approximation of the loss function.
The proposed formulation highlights the importance of staying within the
pre-training basin for achieving composable modules. Moreover, it provides the
basis for two dual incremental training algorithms: the one from the
perspective of multiple models trained individually, while the other aims to
optimize the composed model as a whole. We probe their application in
incremental classification tasks and highlight some valuable skills. In fact,
the pool of incrementally learned modules not only supports the creation of an
effective multi-task model but also enables unlearning and specialization in
specific tasks.
