  A common way to use large pre-trained language models for downstream tasks is
to fine tune them using additional layers. This may not work well if downstream
domain is a specialized domain whereas the large language model has been
pre-trained on a generic corpus. In this paper, we discuss the use of regular
expression patterns employed as features for domain knowledge during the
process of fine tuning, in addition to domain specific text. Our experiments on
real scenario production data show that this method of fine tuning improves the
downstream text classification tasks as compared to fine tuning only on domain
specific text. We also show that the use of attention network for fine tuning
improves results compared to simple linear layers.
