  Language models are pre-trained using large corpora of generic data like book
corpus, common crawl and Wikipedia, which is essential for the model to
understand the linguistic characteristics of the language. New studies suggest
using Domain Adaptive Pre-training (DAPT) and Task-Adaptive Pre-training (TAPT)
as an intermediate step before the final finetuning task. This step helps cover
the target domain vocabulary and improves the model performance on the
downstream task. In this work, we study the impact of training only the
embedding layer on the model's performance during TAPT and task-specific
finetuning. Based on our study, we propose a simple approach to make the
intermediate step of TAPT for BERT-based models more efficient by performing
selective pre-training of BERT layers. We show that training only the BERT
embedding layer during TAPT is sufficient to adapt to the vocabulary of the
target domain and achieve comparable performance. Our approach is
computationally efficient, with 78\% fewer parameters trained during TAPT. The
proposed embedding layer finetuning approach can also be an efficient domain
adaptation technique.
