  Transfer learning (TL) in natural language processing (NLP) has seen a surge
of interest in recent years, as pre-trained models have shown an impressive
ability to transfer to novel tasks. Three main strategies have emerged for
making use of multiple supervised datasets during fine-tuning: training on an
intermediate task before training on the target task (STILTs), using multi-task
learning (MTL) to train jointly on a supplementary task and the target task
(pairwise MTL), or simply using MTL to train jointly on all available datasets
(MTL-ALL). In this work, we compare all three TL methods in a comprehensive
analysis on the GLUE dataset suite. We find that there is a simple heuristic
for when to use one of these techniques over the other: pairwise MTL is better
than STILTs when the target task has fewer instances than the supporting task
and vice versa. We show that this holds true in more than 92% of applicable
cases on the GLUE dataset and validate this hypothesis with experiments varying
dataset size. The simplicity and effectiveness of this heuristic is surprising
and warrants additional exploration by the TL community. Furthermore, we find
that MTL-ALL is worse than the pairwise methods in almost every case. We hope
this study will aid others as they choose between TL methods for NLP tasks.
