  Empathy plays a pivotal role in fostering prosocial behavior, often triggered
by the sharing of personal experiences through narratives. However, modeling
empathy using NLP approaches remains challenging due to its deep
interconnection with human interaction dynamics. Previous approaches, which
involve fine-tuning language models (LMs) on human-annotated empathic datasets,
have had limited success. In our pursuit of improving empathy understanding in
LMs, we propose several strategies, including contrastive learning with masked
LMs and supervised fine-tuning with Large Language Models (LLMs). While these
methods show improvements over previous methods, the overall results remain
unsatisfactory. To better understand this trend, we performed an analysis which
reveals a low agreement among annotators. This lack of consensus hinders
training and highlights the subjective nature of the task. We also explore the
cultural impact on annotations. To study this, we meticulously collected story
pairs in Urdu language and find that subjectivity in interpreting empathy among
annotators appears to be independent of cultural background. The insights from
our systematic exploration of LMs' understanding of empathy suggest that there
is considerable room for exploration in both task formulation and modeling.
