  The therapeutic working alliance is a critical factor in predicting the
success of psychotherapy treatment. Traditionally, working alliance assessment
relies on questionnaires completed by both therapists and patients. In this
paper, we present COMPASS, a novel framework to directly infer the therapeutic
working alliance from the natural language used in psychotherapy sessions. Our
approach utilizes advanced large language models to analyze transcripts of
psychotherapy sessions and compare them with distributed representations of
statements in the working alliance inventory. Analyzing a dataset of over 950
sessions covering diverse psychiatric conditions, we demonstrate the
effectiveness of our method in microscopically mapping patient-therapist
alignment trajectories and providing interpretability for clinical psychiatry
and in identifying emerging patterns related to the condition being treated. By
employing various neural topic modeling techniques in combination with
generative language prompting, we analyze the topical characteristics of
different psychiatric conditions and incorporate temporal modeling to capture
the evolution of topics at a turn-level resolution. This combined framework
enhances the understanding of therapeutic interactions, enabling timely
feedback for therapists regarding conversation quality and providing
interpretable insights to improve the effectiveness of psychotherapy.
