  Large Language Models (LLMs), such as GPT-4 and Llama 2, show remarkable
proficiency in a wide range of natural language processing (NLP) tasks. Despite
their effectiveness, the high costs associated with their use pose a challenge.
We present LlamBERT, a hybrid approach that leverages LLMs to annotate a small
subset of large, unlabeled databases and uses the results for fine-tuning
transformer encoders like BERT and RoBERTa. This strategy is evaluated on two
diverse datasets: the IMDb review dataset and the UMLS Meta-Thesaurus. Our
results indicate that the LlamBERT approach slightly compromises on accuracy
while offering much greater cost-effectiveness.
