  While most previous work has focused on different pretraining objectives and
architectures for transfer learning, we ask how to best adapt the pretrained
model to a given target task. We focus on the two most common forms of
adaptation, feature extraction (where the pretrained weights are frozen), and
directly fine-tuning the pretrained model. Our empirical results across diverse
NLP tasks with two state-of-the-art models show that the relative performance
of fine-tuning vs. feature extraction depends on the similarity of the
pretraining and target tasks. We explore possible explanations for this finding
and provide a set of adaptation guidelines for the NLP practitioner.
