  Prevalent supervised learning methods in natural language processing (NLP)
are notoriously data-hungry, which demand large amounts of high-quality
annotated data. In practice, acquiring such data is a costly endeavor.
Recently, the superior few-shot performance of large language models (LLMs) has
propelled the development of dataset generation, where the training data are
solely synthesized from LLMs. However, such an approach usually suffers from
low-quality issues, and requires orders of magnitude more labeled data to
achieve satisfactory performance. To fully exploit the potential of LLMs and
make use of massive unlabeled data, we propose LLMaAA, which takes LLMs as
annotators and puts them into an active learning loop to determine what to
annotate efficiently. To learn robustly with pseudo labels, we optimize both
the annotation and training processes: (1) we draw k-NN examples from a small
demonstration pool as in-context examples, and (2) we adopt the example
reweighting technique to assign training samples with learnable weights.
Compared with previous approaches, LLMaAA features both efficiency and
reliability. We conduct experiments and analysis on two classic NLP tasks,
named entity recognition and relation extraction. With LLMaAA, task-specific
models trained from LLM-generated labels can outperform the teacher within only
hundreds of annotated examples, which is much more cost-effective than other
baselines.
