  This paper presents a novel framework for quantitatively evaluating the
interactive ChatGPT model in the context of suicidality assessment from social
media posts, utilizing the University of Maryland Reddit suicidality dataset.
We conduct a technical evaluation of ChatGPT's performance on this task using
Zero-Shot and Few-Shot experiments and compare its results with those of two
fine-tuned transformer-based models. Additionally, we investigate the impact of
different temperature parameters on ChatGPT's response generation and discuss
the optimal temperature based on the inconclusiveness rate of ChatGPT. Our
results indicate that while ChatGPT attains considerable accuracy in this task,
transformer-based models fine-tuned on human-annotated datasets exhibit
superior performance. Moreover, our analysis sheds light on how adjusting the
ChatGPT's hyperparameters can improve its ability to assist mental health
professionals in this critical task.
