  This paper studies the performance of open-source Large Language Models
(LLMs) in text classification tasks typical for political science research. By
examining tasks like stance, topic, and relevance classification, we aim to
guide scholars in making informed decisions about their use of LLMs for text
analysis. Specifically, we conduct an assessment of both zero-shot and
fine-tuned LLMs across a range of text annotation tasks using news articles and
tweets datasets. Our analysis shows that fine-tuning improves the performance
of open-source LLMs, allowing them to match or even surpass zero-shot GPT-3.5
and GPT-4, though still lagging behind fine-tuned GPT-3.5. We further establish
that fine-tuning is preferable to few-shot training with a relatively modest
quantity of annotated text. Our findings show that fine-tuned open-source LLMs
can be effectively deployed in a broad spectrum of text annotation
applications. We provide a Python notebook facilitating the application of LLMs
in text annotation for other researchers.
