  Warning: this paper contains model outputs exhibiting offensiveness and
biases. Recently pre-trained language models (PLMs) have prospered in various
natural language generation (NLG) tasks due to their ability to generate fairly
fluent text. Nevertheless, these models are observed to capture and reproduce
harmful contents in training corpora, typically toxic language and social
biases, raising severe moral issues. Prior works on ethical NLG tackle
detoxifying and debiasing separately, which is problematic since we find
debiased models still exhibit toxicity while detoxified ones even exacerbate
social biases. To address such a challenge, we propose the first unified
framework of detoxifying and debiasing called UDDIA, which jointly formalizes
these two problems as rectifying the output space. We theoretically interpret
our framework as learning a text distribution mixing weighted attributes.
Besides, UDDIA conducts adaptive optimization of only a few parameters during
decoding based on a parameter-efficient tuning schema without any training
data. This leads to minimal generation quality loss and improved rectification
performance with acceptable computational cost. Experimental results
demonstrate that compared to several strong baselines, UDDIA achieves debiasing
and detoxifying simultaneously and better balances efficiency and
effectiveness, taking a further step towards practical ethical NLG.
