  Existing studies have investigated the tendency of autoregressive language
models to generate contexts that exhibit undesired biases and toxicity. Various
debiasing approaches have been proposed, which are primarily categorized into
data-based and decoding-based. In our study, we investigate the ensemble of the
two debiasing paradigms, proposing to use toxic corpus as an additional
resource to reduce the toxicity. Our result shows that toxic corpus can indeed
help to reduce the toxicity of the language generation process substantially,
complementing the existing debiasing methods.
