  The release of ChatGPT has uncovered a range of possibilities whereby large
language models (LLMs) can substitute human intelligence. In this paper, we
seek to understand whether ChatGPT has the potential to reproduce
human-generated label annotations in social computing tasks. Such an
achievement could significantly reduce the cost and complexity of social
computing research. As such, we use ChatGPT to relabel five seminal datasets
covering stance detection (2x), sentiment analysis, hate speech, and bot
detection. Our results highlight that ChatGPT does have the potential to handle
these data annotation tasks, although a number of challenges remain. ChatGPT
obtains an average accuracy 0.609. Performance is highest for the sentiment
analysis dataset, with ChatGPT correctly annotating 64.9% of tweets. Yet, we
show that performance varies substantially across individual labels. We believe
this work can open up new lines of analysis and act as a basis for future
research into the exploitation of ChatGPT for human annotation tasks.
