  Realistic practice and tailored feedback are key processes for training peer
counselors with clinical skills. However, existing mechanisms of providing
feedback largely rely on human supervision. Peer counselors often lack
mechanisms to receive detailed feedback from experienced mentors, making it
difficult for them to support the large number of people with mental health
issues who use peer counseling. Our work aims to leverage large language models
to provide contextualized and multi-level feedback to empower peer counselors,
especially novices, at scale. To achieve this, we co-design with a group of
senior psychotherapy supervisors to develop a multi-level feedback taxonomy,
and then construct a publicly available dataset with comprehensive feedback
annotations of 400 emotional support conversations. We further design a
self-improvement method on top of large language models to enhance the
automatic generation of feedback. Via qualitative and quantitative evaluation
with domain experts, we demonstrate that our method minimizes the risk of
potentially harmful and low-quality feedback generation which is desirable in
such high-stakes scenarios.
