  Large Language Models (LLMs) are quickly becoming ubiquitous, but the
implications for social science research are not yet well understood. This
paper asks whether LLMs can help us analyse large-N qualitative data from
open-ended interviews, with an application to transcripts of interviews with
Rohingya refugees in Cox's Bazaar, Bangladesh. We find that a great deal of
caution is needed in using LLMs to annotate text as there is a risk of
introducing biases that can lead to misleading inferences. We here mean bias in
the technical sense, that the errors that LLMs make in annotating interview
transcripts are not random with respect to the characteristics of the interview
subjects. Training simpler supervised models on high-quality human annotations
with flexible coding leads to less measurement error and bias than LLM
annotations. Therefore, given that some high quality annotations are necessary
in order to asses whether an LLM introduces bias, we argue that it is probably
preferable to train a bespoke model on these annotations than it is to use an
LLM for annotation.
