  Empathetic response generation, aiming at understanding the user's situation
and feelings and respond empathically, is crucial in building human-like
dialogue systems. Previous methods mainly focus on using maximum likelihood
estimation as the optimization objective for training response generation
models, without taking into account the empathy level alignment between
generated responses and target responses. To this end, we propose an empathetic
response generation using reinforcement learning (EmpRL) framework. The
framework designs an effective empathy reward function and generates empathetic
responses by maximizing the expected reward through reinforcement learning.
Given the powerful text generation capability of pre-trained language models,
EmpRL utilizes the pre-trained T5 model as the generator and conducts further
training to initialize the policy. To align the empathy level between generated
responses and target responses in the context, an empathy reward function
containing three empathy communication mechanisms, i.e., emotional reaction,
interpretation, and exploration, is constructed using pre-designed and
pre-trained empathy identifiers. Finally, the proximal policy optimization
algorithm is used to further train the policy to produce empathetic responses.
Both automatic and manual evaluations demonstrate that the proposed EmpRL
framework can improve the quality of generated responses, enhance the empathy
level similarity between generated and target responses, and produce empathetic
responses covering both affective and cognitive aspects.
