  Neural Transfer Learning (TL) is becoming ubiquitous in Natural Language
Processing (NLP), thanks to its high performance on many tasks, especially in
low-resourced scenarios. Notably, TL is widely used for neural domain
adaptation to transfer valuable knowledge from high-resource to low-resource
domains. In the standard fine-tuning scheme of TL, a model is initially
pre-trained on a source domain and subsequently fine-tuned on a target domain
and, therefore, source and target domains are trained using the same
architecture. In this paper, we show through interpretation methods that such
scheme, despite its efficiency, is suffering from a main limitation. Indeed,
although capable of adapting to new domains, pre-trained neurons struggle with
learning certain patterns that are specific to the target domain. Moreover, we
shed light on the hidden negative transfer occurring despite the high
relatedness between source and target domains, which may mitigate the final
gain brought by transfer learning. To address these problems, we propose to
augment the pre-trained model with normalised, weighted and randomly
initialised units that foster a better adaptation while maintaining the
valuable source knowledge. We show that our approach exhibits significant
improvements to the standard fine-tuning scheme for neural domain adaptation
from the news domain to the social media domain on four NLP tasks:
part-of-speech tagging, chunking, named entity recognition and morphosyntactic
tagging.
