  Recent studies have demonstrated promising potential of ChatGPT for various
text annotation and classification tasks. However, ChatGPT is non-deterministic
which means that, as with human coders, identical input can lead to different
outputs. Given this, it seems appropriate to test the reliability of ChatGPT.
Therefore, this study investigates the consistency of ChatGPT's zero-shot
capabilities for text annotation and classification, focusing on different
model parameters, prompt variations, and repetitions of identical inputs. Based
on the real-world classification task of differentiating website texts into
news and not news, results show that consistency in ChatGPT's classification
output can fall short of scientific thresholds for reliability. For example,
even minor wording alterations in prompts or repeating the identical input can
lead to varying outputs. Although pooling outputs from multiple repetitions can
improve reliability, this study advises caution when using ChatGPT for
zero-shot text annotation and underscores the need for thorough validation,
such as comparison against human-annotated data. The unsupervised application
of ChatGPT for text annotation and classification is not recommended.
