  Transfer learning techniques are particularly useful in NLP tasks where a
sizable amount of high-quality annotated data is difficult to obtain. Current
approaches directly adapt a pre-trained language model (LM) on in-domain text
before fine-tuning to downstream tasks. We show that extending the vocabulary
of the LM with domain-specific terms leads to further gains. To a bigger
effect, we utilize structure in the unlabeled data to create auxiliary
synthetic tasks, which helps the LM transfer to downstream tasks. We apply
these approaches incrementally on a pre-trained Roberta-large LM and show
considerable performance gain on three tasks in the IT domain: Extractive
Reading Comprehension, Document Ranking and Duplicate Question Detection.
