  Fine-tuning neural networks is widely used to transfer valuable knowledge
from high-resource to low-resource domains. In a standard fine-tuning scheme,
source and target problems are trained using the same architecture. Although
capable of adapting to new domains, pre-trained units struggle with learning
uncommon target-specific patterns. In this paper, we propose to augment the
target-network with normalised, weighted and randomly initialised units that
beget a better adaptation while maintaining the valuable source knowledge. Our
experiments on POS tagging of social media texts (Tweets domain) demonstrate
that our method achieves state-of-the-art performances on 3 commonly used
datasets.
