  Large language models (LLMs) are already being piloted for clinical use in
hospital systems like NYU Langone, Dana-Farber and the NHS. A proposed
deployment use case is psychotherapy, where a LLM-powered chatbot can treat a
patient undergoing a mental health crisis. Deployment of LLMs for mental health
response could hypothetically broaden access to psychotherapy and provide new
possibilities for personalizing care. However, recent high-profile failures,
like damaging dieting advice offered by the Tessa chatbot to patients with
eating disorders, have led to doubt about their reliability in high-stakes and
safety-critical settings.
  In this work, we develop an evaluation framework for determining whether LLM
response is a viable and ethical path forward for the automation of mental
health treatment. Using human evaluation with trained clinicians and automatic
quality-of-care metrics grounded in psychology research, we compare the
responses provided by peer-to-peer responders to those provided by a
state-of-the-art LLM.
  We show that LLMs like GPT-4 use implicit and explicit cues to infer patient
demographics like race. We then show that there are statistically significant
discrepancies between patient subgroups: Responses to Black posters
consistently have lower empathy than for any other demographic group (2%-13%
lower than the control group). Promisingly, we do find that the manner in which
responses are generated significantly impacts the quality of the response. We
conclude by proposing safety guidelines for the potential deployment of LLMs
for mental health response.
