  Transfer learning from large language models (LLMs) has emerged as a powerful
technique to enable knowledge-based fine-tuning for a number of tasks,
adaptation of models for different domains and even languages. However, it
remains an open question, if and when transfer learning will work, i.e. leading
to positive or negative transfer. In this paper, we analyze the knowledge
transfer across three natural language processing (NLP) tasks - text
classification, sentimental analysis, and sentence similarity, using three LLMs
- BERT, RoBERTa, and XLNet - and analyzing their performance, by fine-tuning on
target datasets for domain and cross-lingual adaptation tasks, with and without
an intermediate task training on a larger dataset. Our experiments showed that
fine-tuning without an intermediate task training can lead to a better
performance for most tasks, while more generalized tasks might necessitate a
preceding intermediate task training step. We hope that this work will act as a
guide on transfer learning to NLP practitioners.
