  Keyword extraction has been an important topic for modern natural language
processing. With its applications ranging from ontology generation, fact
verification in summarized text, and recommendation systems. While it has had
significant data-intensive applications, it is often hampered when the data set
is small. Downstream training for keyword extractors is a lengthy process and
requires a significant amount of data. Recently, Few-shot Learning (FSL) and
Zero-Shot Learning (ZSL) have been proposed to tackle this problem. Therefore,
we propose AdaptKeyBERT, a pipeline for training keyword extractors with LLM
bases by incorporating the concept of regularized attention into a pre-training
phase for downstream domain adaptation. As we believe our work has implications
to be utilized in the pipeline of FSL/ZSL and keyword extraction, we
open-source our code as well as provide the fine-tuning library of the same
name AdaptKeyBERT at https://github.com/AmanPriyanshu/AdaptKeyBERT.
