  Transformer-based language models are able to generate fluent text and be
efficiently adapted across various natural language generation tasks. However,
language models that are pretrained on large unlabeled web text corpora have
been shown to suffer from degenerating toxic content and social bias behaviors,
consequently hindering their safe deployment. Various detoxification methods
were proposed to mitigate the language model's toxicity; however, these methods
struggled to detoxify language models when conditioned on prompts that contain
specific social identities related to gender, race, or religion. In this study,
we propose Reinforce-Detoxify; A reinforcement learning-based method for
mitigating toxicity in language models. We address the challenge of safety in
language models and propose a new reward model that is able to detect toxic
content and mitigate unintended bias towards social identities in toxicity
prediction. The experiments demonstrate that the Reinforce-Detoxify method for
language model detoxification outperforms existing detoxification approaches in
automatic evaluation metrics, indicating the ability of our approach in
language model detoxification and less prone to unintended bias toward social
identities in generated content.
