  During the fine-tuning phase of transfer learning, the pretrained vocabulary
remains unchanged, while model parameters are updated. The vocabulary generated
based on the pretrained data is suboptimal for downstream data when domain
discrepancy exists. We propose to consider the vocabulary as an optimizable
parameter, allowing us to update the vocabulary by expanding it with
domain-specific vocabulary based on a tokenization statistic. Furthermore, we
preserve the embeddings of the added words from overfitting to downstream data
by utilizing knowledge learned from a pretrained language model with a
regularization term. Our method achieved consistent performance improvements on
diverse domains (i.e., biomedical, computer science, news, and reviews).
