  Amidst the growing interest in developing task-autonomous AI for automated
mental health care, this paper addresses the ethical and practical challenges
associated with the issue and proposes a structured framework that delineates
levels of autonomy, outlines ethical requirements, and defines beneficial
default behaviors for AI agents in the context of mental health support. We
also evaluate ten state-of-the-art language models using 16 mental
health-related questions designed to reflect various mental health conditions,
such as psychosis, mania, depression, suicidal thoughts, and homicidal
tendencies. The question design and response evaluations were conducted by
mental health clinicians (M.D.s). We find that existing language models are
insufficient to match the standard provided by human professionals who can
navigate nuances and appreciate context. This is due to a range of issues,
including overly cautious or sycophantic responses and the absence of necessary
safeguards. Alarmingly, we find that most of the tested models could cause harm
if accessed in mental health emergencies, failing to protect users and
potentially exacerbating existing symptoms. We explore solutions to enhance the
safety of current models. Before the release of increasingly task-autonomous AI
systems in mental health, it is crucial to ensure that these models can
reliably detect and manage symptoms of common psychiatric disorders to prevent
harm to users. This involves aligning with the ethical framework and default
behaviors outlined in our study. We contend that model developers are
responsible for refining their systems per these guidelines to safeguard
against the risks posed by current AI technologies to user mental health and
safety.
  Trigger warning: Contains and discusses examples of sensitive mental health
topics, including suicide and self-harm.
