  Collecting labeled datasets in finance is challenging due to scarcity of
domain experts and higher cost of employing them. While Large Language Models
(LLMs) have demonstrated remarkable performance in data annotation tasks on
general domain datasets, their effectiveness on domain specific datasets
remains underexplored. To address this gap, we investigate the potential of
LLMs as efficient data annotators for extracting relations in financial
documents. We compare the annotations produced by three LLMs (GPT-4, PaLM 2,
and MPT Instruct) against expert annotators and crowdworkers. We demonstrate
that the current state-of-the-art LLMs can be sufficient alternatives to
non-expert crowdworkers. We analyze models using various prompts and parameter
settings and find that customizing the prompts for each relation group by
providing specific examples belonging to those groups is paramount.
Furthermore, we introduce a reliability index (LLM-RelIndex) used to identify
outputs that may require expert attention. Finally, we perform an extensive
time, cost and error analysis and provide recommendations for the collection
and usage of automated annotations in domain-specific settings.
