  Deep pretrained language models have achieved great success in the way of
pretraining first and then fine-tuning. But such a sequential transfer learning
paradigm often confronts the catastrophic forgetting problem and leads to
sub-optimal performance. To fine-tune with less forgetting, we propose a recall
and learn mechanism, which adopts the idea of multi-task learning and jointly
learns pretraining tasks and downstream tasks. Specifically, we propose a
Pretraining Simulation mechanism to recall the knowledge from pretraining tasks
without data, and an Objective Shifting mechanism to focus the learning on
downstream tasks gradually. Experiments show that our method achieves
state-of-the-art performance on the GLUE benchmark. Our method also enables
BERT-base to achieve better performance than directly fine-tuning of
BERT-large. Further, we provide the open-source RecAdam optimizer, which
integrates the proposed mechanisms into Adam optimizer, to facility the NLP
community.
