  A growing number of state-of-the-art transfer learning methods employ
language models pretrained on large generic corpora. In this paper we present a
conceptually simple and effective transfer learning approach that addresses the
problem of catastrophic forgetting. Specifically, we combine the task-specific
optimization function with an auxiliary language model objective, which is
adjusted during the training process. This preserves language regularities
captured by language models, while enabling sufficient adaptation for solving
the target task. Our method does not require pretraining or finetuning separate
components of the network and we train our models end-to-end in a single step.
We present results on a variety of challenging affective and text
classification tasks, surpassing well established transfer learning methods
with greater level of complexity.
