  In this study, we address one of the challenges of developing NER models for
scholarly domains, namely the scarcity of suitable labeled data. We experiment
with an approach using predictions from a fine-tuned LLM model to aid
non-domain experts in annotating scientific entities within astronomy
literature, with the goal of uncovering whether such a collaborative process
can approximate domain expertise. Our results reveal moderate agreement between
a domain expert and the LLM-assisted non-experts, as well as fair agreement
between the domain expert and the LLM model's predictions. In an additional
experiment, we compare the performance of finetuned and default LLMs on this
task. We have also introduced a specialized scientific entity annotation scheme
for astronomy, validated by a domain expert. Our approach adopts a scholarly
research contribution-centric perspective, focusing exclusively on scientific
entities relevant to the research theme. The resultant dataset, containing
5,000 annotated astronomy article titles, is made publicly available.
