  Sequential fine-tuning and multi-task learning are methods aiming to
incorporate knowledge from multiple tasks; however, they suffer from
catastrophic forgetting and difficulties in dataset balancing. To address these
shortcomings, we propose AdapterFusion, a new two stage learning algorithm that
leverages knowledge from multiple tasks. First, in the knowledge extraction
stage we learn task specific parameters called adapters, that encapsulate the
task-specific information. We then combine the adapters in a separate knowledge
composition step. We show that by separating the two stages, i.e., knowledge
extraction and knowledge composition, the classifier can effectively exploit
the representations learned from multiple tasks in a non-destructive manner. We
empirically evaluate AdapterFusion on 16 diverse NLU tasks, and find that it
effectively combines various types of knowledge at different layers of the
model. We show that our approach outperforms traditional strategies such as
full fine-tuning as well as multi-task learning. Our code and adapters are
available at AdapterHub.ml.
