  Bias in LLMs can harm user experience and societal outcomes. However, current
bias mitigation methods often require intensive human feedback, lack
transferability to other topics or yield overconfident and random outputs. We
find that involving LLMs in role-playing scenario boosts their ability to
recognize and mitigate biases. Based on this, we propose Reinforcement Learning
from Multi-role Debates as Feedback (RLDF), a novel approach for bias
mitigation replacing human feedback in traditional RLHF. We utilize LLMs in
multi-role debates to create a dataset that includes both high-bias and
low-bias instances for training the reward model in reinforcement learning. Our
approach comprises two modes: (1) self-reflection, where the same LLM
participates in multi-role debates, and (2) teacher-student, where a more
advanced LLM like GPT-3.5-turbo guides the LLM to perform this task.
Experimental results across different LLMs demonstrate the effectiveness of our
approach in bias mitigation.
