  Large Language Models (LLMs) operating in 0-shot or few-shot settings achieve
competitive results in Text Classification tasks. In-Context Learning (ICL)
typically achieves better accuracy than the 0-shot setting, but it pays in
terms of efficiency, due to the longer input prompt. In this paper, we propose
a strategy to make LLMs as efficient as 0-shot text classifiers, while getting
comparable or better accuracy than ICL. Our solution targets the low resource
setting, i.e., when only 4 examples per class are available. Using a single LLM
and few-shot real data we perform a sequence of generation, filtering and
Parameter-Efficient Fine-Tuning steps to create a robust and efficient
classifier. Experimental results show that our approach leads to competitive
results on multiple text classification datasets.
